<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<meta http-equiv="cache-control" content="no-cache">
<title>RE/flex user guide</title>
<link href="doxygen_tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css">
<link href="doxygen_content.css" rel="stylesheet" type="text/css">
</head>
<body>
<div id="top">
 <div id="titlearea">
  <table height="72px" width="100%" cellspacing="0" cellpadding="0">
   <tbody>
    <tr>
     <td width="10%">&nbsp;</td>
     <td><a href="https://github.com/Genivia/RE-flex"><img src="reflex-logo.png"/></a></td>
     <td>
      <div style="float: right; font-size: 18px; font-weight: bold;">RE/flex user guide</div>
      <br>
      <div style="float: right; font-size: 10px;">updated Thu May 14 2020 by Robert van Engelen</div>
     </td>
     <td width="10%">&nbsp;</td>
    </tr>
   </tbody>
  </table>
 </div>
<!-- Generated by Doxygen 1.8.11 -->
  <div id="navrow1" class="tabs">
    <ul class="tablist">
      <li class="current"><a href="index.html"><span>Main&#160;Page</span></a></li>
      <li><a href="namespaces.html"><span>Namespaces</span></a></li>
      <li><a href="annotated.html"><span>Classes</span></a></li>
      <li><a href="files.html"><span>Files</span></a></li>
    </ul>
  </div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">RE/flex user guide </div>  </div>
</div><!--header-->
<div class="contents">
<div class="toc"><h3>Table of Contents</h3>
<ul><li class="level1"><a href="#intro">What is RE/flex?                                                       </a><ul><li class="level2"><a href="#intro1">Yet another scanner generator                                         </a></li>
<li class="level2"><a href="#intro2">And a flexible regex library                                          </a></li>
</ul>
</li>
<li class="level1"><a href="#reflex">The RE/flex scanner generator                                         </a><ul><li class="level2"><a href="#reflex-command">The reflex command line tool                                  </a></li>
<li class="level2"><a href="#reflex-options">Command-line options                                          </a><ul><li class="level3"><a href="#reflex-options-scanner">Scanner options</a></li>
<li class="level3"><a href="#reflex-options-files">Output files options</a></li>
<li class="level3"><a href="#reflex-options-code">Output code options</a></li>
<li class="level3"><a href="#reflex-options-debug">Debugging options</a></li>
<li class="level3"><a href="#reflex-options-miscellaneous">Miscellaneous options</a></li>
</ul>
</li>
<li class="level2"><a href="#reflex-spec">Lexer specifications                                             </a><ul><li class="level3"><a href="#reflex-spec-defs">The definitions section</a></li>
<li class="level3"><a href="#reflex-spec-rules">The rules section</a></li>
<li class="level3"><a href="#reflex-spec-user">User code sections</a></li>
</ul>
</li>
<li class="level2"><a href="#reflex-patterns">Patterns                                                     </a><ul><li class="level3"><a href="#reflex-pattern-syntax">Pattern syntax</a></li>
<li class="level3"><a href="#reflex-pattern-class">Character classes</a></li>
<li class="level3"><a href="#reflex-pattern-cat">Character categories</a></li>
<li class="level3"><a href="#reflex-pattern-anchor">Anchors and boundaries</a></li>
<li class="level3"><a href="#reflex-pattern-dents">Indent/nodent/dedent</a></li>
<li class="level3"><a href="#reflex-pattern-negative">Negative patterns</a></li>
<li class="level3"><a href="#reflex-pattern-lookahead">Lookahead</a></li>
<li class="level3"><a href="#reflex-pattern-lookbehind">Lookbehind</a></li>
<li class="level3"><a href="#reflex-pattern-trailing">Trailing context</a></li>
<li class="level3"><a href="#reflex-pattern-unicode">Unicode mode</a></li>
<li class="level3"><a href="#reflex-pattern-freespace">Free space mode</a></li>
<li class="level3"><a href="#reflex-pattern-multiline">Multi-line mode</a></li>
<li class="level3"><a href="#reflex-pattern-dotall">Dotall mode</a></li>
<li class="level3"><a href="#reflex-pattern-anycase">Case-insensitive mode</a></li>
<li class="level3"><a href="#reflex-pattern-modifiers">Multiple mode modifiers</a></li>
</ul>
</li>
<li class="level2"><a href="#reflex-lexer">The Lexer/yyFlexLexer class                                     </a></li>
<li class="level2"><a href="#reflex-inherit">Inheriting Lexer/yyFlexLexer                                  </a></li>
<li class="level2"><a href="#reflex-multiple">Combining multiple lexers                                    </a></li>
<li class="level2"><a href="#reflex-input">Switching input sources                                         </a></li>
<li class="level2"><a href="#reflex-multiple-input">Multiple input sources                                 </a></li>
<li class="level2"><a href="#reflex-states">Start condition states                                         </a></li>
<li class="level2"><a href="#reflex-code-blocks">Initial code blocks                                       </a></li>
<li class="level2"><a href="#reflex-bison">Interfacing with Bison/Yacc                                     </a><ul><li class="level3"><a href="#reflex-bison-mt-safe">Bison and thread-safety</a></li>
<li class="level3"><a href="#reflex-bison-cc">Bison-cc</a></li>
<li class="level3"><a href="#reflex-bison-cc-locations">Bison-cc &amp; locations</a></li>
<li class="level3"><a href="#reflex-bison-complete">Bison-complete</a></li>
<li class="level3"><a href="#reflex-bison-complete-locations">Bison-complete &amp; locations</a></li>
<li class="level3"><a href="#reflex-bison-bridge">Bison-bridge</a></li>
<li class="level3"><a href="#reflex-bison-locations">Bison-locations</a></li>
<li class="level3"><a href="#reflex-bison-bridge-locations">Bison-bridge &amp; locations</a></li>
<li class="level3"><a href="#reflex-reentrant">Reentrant scanners</a></li>
</ul>
</li>
<li class="level2"><a href="#reflex-search">Searching versus scanning                                      </a></li>
<li class="level2"><a href="#reflex-posix-perl">POSIX versus Perl matching                                 </a></li>
<li class="level2"><a href="#reflex-debug">Debugging and profiling                                         </a></li>
<li class="level2"><a href="#reflex-examples">Examples                                                     </a></li>
<li class="level2"><a href="#reflex-limitations">Limitations                                               </a></li>
</ul>
</li>
<li class="level1"><a href="#regex">The RE/flex regex library                                              </a><ul><li class="level2"><a href="#regex-boost">Boost matcher classes                                            </a></li>
<li class="level2"><a href="#regex-pcre2">PCRE2 matcher classes                                            </a></li>
<li class="level2"><a href="#regex-std">std::regex matcher classes                                         </a></li>
<li class="level2"><a href="#regex-matcher">The reflex::Matcher class                                      </a></li>
<li class="level2"><a href="#regex-pattern">The reflex::Pattern class                                      </a></li>
<li class="level2"><a href="#regex-convert">Regex converters                                               </a></li>
<li class="level2"><a href="#regex-methods">Methods and iterators                                          </a><ul><li class="level3"><a href="#regex-methods-matches">matches</a></li>
<li class="level3"><a href="#regex-methods-find">find</a></li>
<li class="level3"><a href="#regex-methods-scan">scan</a></li>
<li class="level3"><a href="#regex-methods-split">split</a></li>
<li class="level3"><a href="#regex-methods-props">Properties of a match</a></li>
<li class="level3"><a href="#regex-members">Public data members</a></li>
<li class="level3"><a href="#regex-methods-pattern">Pattern methods</a></li>
<li class="level3"><a href="#regex-methods-input">Input methods</a></li>
</ul>
</li>
<li class="level2"><a href="#regex-input">The Input class                                                  </a><ul><li class="level3"><a href="#regex-input-assign">Assigning input</a></li>
<li class="level3"><a href="#regex-input-strings">Input strings</a></li>
<li class="level3"><a href="#regex-input-streams">Input streams</a></li>
<li class="level3"><a href="#regex-input-file">FILE encodings</a></li>
<li class="level3"><a href="#regex-input-properties">Input properties</a></li>
<li class="level3"><a href="#regex-input-streambuf">Input streambuf</a></li>
<li class="level3"><a href="#regex-input-dosstreambuf">DOS CRLF newlines</a></li>
</ul>
</li>
<li class="level2"><a href="#regex-examples">Examples                                                      </a></li>
</ul>
</li>
<li class="level1"><a href="#tricks">Tips, tricks, and gotchas                                             </a><ul><li class="level2"><a href="#extern-yyin">Errors when declaring extern yyin, yytext, yylineno              </a></li>
<li class="level2"><a href="#invalid-utf">Invalid UTF encodings                                            </a></li>
<li class="level2"><a href="#errors">Error reporting and recovery                                          </a></li>
<li class="level2"><a href="#setlocale">On using setlocale                                                 </a></li>
<li class="level2"><a href="#iso-8859-1">Scanning ISO-8859-1 (latin-1) files with a Unicode scanner        </a></li>
<li class="level2"><a href="#crlf">Handling DOS CRLF newlines                                              </a></li>
<li class="level2"><a href="#cr">Handling old Macintosh files containing CR newlines                       </a></li>
<li class="level2"><a href="#lazy">Lazy repetitions                                                        </a></li>
<li class="level2"><a href="#trigraphs">Lazy optional patterns and trigraphs                               </a></li>
<li class="level2"><a href="#switching">Repeately switching to the same input                              </a></li>
<li class="level2"><a href="#flexlexer">Where is FlexLexer.h?                                              </a></li>
<li class="level2"><a href="#readline">Interactive input with GNU readline                                 </a></li>
<li class="level2"><a href="#link-errors">Undefined symbols and link errors                                </a></li>
<li class="level2"><a href="#linking">Minimized library                                                    </a></li>
<li class="level2"><a href="#msvc">MSVC++ compiler bug                                                     </a></li>
</ul>
</li>
<li class="level1"><a href="#bugs">Bugs                                                                    </a></li>
<li class="level1"><a href="#download">Installing RE/flex                                                  </a></li>
<li class="level1"><a href="#license">License and copyright                                                </a></li>
</ul>
</div>
<div class="textblock"><p>"The asteroid to kill this dinosaur is still in orbit." &ndash; Lex Manual</p>
<p>"Optimize: this currently has no effect in Boost.Regex." &ndash; Boost Manual</p>
<p>"Reflex: a thing that is determined by and reproduces
   the essential features or qualities of something else." &ndash; Oxford Dictionary</p>
<h1><a class="anchor" id="intro"></a>
What is RE/flex?                                                       </h1>
<p>RE/flex is the regex-centric, fast lexical analyzer generator with full Unicode support, indentation anchors, lazy quantifiers, word boundaries, and many other modern features. RE/flex also includes a fast regex engine written in C++ with options to generate finite state machine tables or direct code to match input more efficiently. RE/flex includes a smart input class to normalize input from files, streams, strings, and memory. RE/flex is compatible with Bison/Yacc and accepts Flex lexer specifications.</p>
<p>Features:</p>
<ul>
<li>faster than Flex++ for typical applications such as tokenization;</li>
<li>accepts Flex/Lex lexer specifications, extended to support Unicode;</li>
<li>fully supports Unicode, auto-detects UTF-8/16/32 input;</li>
<li>supports legacy file encoding formats, e.g. CP 1250, EBCDIC.</li>
<li>easily integrates with Bison reentrant, C++, bridge and location parsers;</li>
<li>generates source code that is easy to understand;</li>
<li>generates thread-safe scanners;</li>
<li>generates graphviz files for visualization of finite state machines;</li>
<li>options for intuitive customization of the lexer class source code output;</li>
<li>efficient matching in direct code or with finite state machine tables;</li>
<li>optional "free space mode" to improve readability of lexer specifications;</li>
<li>regular expressions may contain lazy quantifiers;</li>
<li>regular expressions may contain word boundary anchors;</li>
<li>regular expressions may contain indent/dedent markers for matching;</li>
<li>other regex engines to choose from, such as PCRE2 and Boost.Regex;</li>
<li>released under a permissive open source license (BSD-3).</li>
</ul>
<p>RE/flex offers lots of other practical improvements over Flex++, such as:</p>
<ul>
<li>no input buffer length limit (Flex has a 16KB limit);</li>
<li><code>yypush_buffer_state</code> saves the scanner state (line, column, and indentation positions), not just the input buffer;</li>
<li>new methods to analyze ASCII and Unicode input, such as <code>str()</code> and <code>wstr()</code> to obtain the (wide) string match, <code>line()</code> and <code>wline()</code> to obtain the current (wide) line for error reporting.</li>
</ul>
<p>Rule patterns in a lexer specification are converted by the <b><code>reflex</code></b> tool to efficient deterministic finite state machines in direct code (option <code>−−fast</code> or in opcode tables <code>−−full</code>. Other regex engines to choose from include PCRE2 and Boost.Regex for Perl and POSIX matching modes. UTF-8/16/32 file input normalization for Unicode pattern matching is performed automatically. Other encodings can be programmatically specified with minimal codeing. Therefore, RE/flex scanners can work on any type of input.</p>
<p>RE/flex incorporates proper object-oriented design principles and does not rely on macros and globals as Flex does. Macros and globals are added to the source code generated by <b><code>reflex</code></b> only when option <code>−−flex</code> is specified. A RE/flex scanner is a generated class derived from a base lexer class template, with its matcher engine defined as a template parameter.</p>
<p>RE/flex is compatible with Lex/Flex and Bison/Yacc with options <code>−−flex</code> and <code>−−bison</code>, respectively. Option <code>−−yy</code> forces basic, no-frills Lex POSIX compliance. RE/flex also offers specific options to seamlessly integrate Bison bridge, Bison locations, Bison C++, Bison complete, and reentrant parsers.</p>
<p>In this document we refer to a <em>regex</em> as a shorthand for <em>regular expression</em>, However, a "regular expression" refers to the formal concept of <em>regular languages</em>, wheras <em>regex</em> often refers to backtracking-based regex matching that Perl introduced. Both concepts are applicable to RE/flex patterns.</p>
<p>In summary, RE/flex is really several things combined into one package:</p>
<ul>
<li>a faster, feature-rich replacement of <a href="dinosaur.compilertools.net/#flex">Flex++ and Flex</a>;</li>
<li>a stand-alone regex library for fast regex matching in C++;</li>
<li>a C++ regex API enhancement of the <a href="www.boost.org/libs/regex">Boost.Regex</a> and <a href="www.pcre.org">PCRE2</a> libraries for matching, seaching, splitting and scanning of input, with input from (wide) strings, files, and streams of potentially unlimited length.</li>
</ul>
<p>The typographical conventions used by this document are:</p>
<ul>
<li><code>Courier</code> denotes C and C++ source code.</li>
<li><em><code>Courier</code></em> denotes lexer specifications and file names.</li>
<li><b><code>Courier</code></b> denotes commands and command or program output displayed in a terminal window.</li>
</ul>
<dl class="section note"><dt>Note</dt><dd>This is a note to clarify a technical matter.</dd></dl>
<dl class="section warning"><dt>Warning</dt><dd>Look out for warnings!</dd></dl>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="intro1"></a>
Yet another scanner generator                                         </h2>
<p>Lex, Flex and variants are powerful <em>scanner generators</em> that generate scanners (a.k.a. <em>lexical analyzers</em> and <em>lexers</em>) from lexer specifications. The lexer specifications define patterns with user-defined actions that are executed when their patterns match the input stream. The scanner repeatedly matches patterns and triggers these actions until the end of the input stream is reached.</p>
<p>Both Lex and Flex are popular to develop <em>tokenizers</em> in which the user-defined actions emit or return a <em>token</em> when the corresponding pattern matches. These tokenizers are typically implemented to scan and tokenize the source code for a compiler or an interpreter of a programming language. The <em>regular expression</em> patterns in a tokenizer define the make-up of identifiers, constants, keywords, punctuation, and to skip over white space in the source code that is scanned.</p>
<p>Consider for example the following patterns and associated actions defined in a lexer specification:</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line"><span class="comment">/* PATTERN */</span>           <span class="comment">/* ACTION */</span></div><div class="line"><span class="stringliteral">&quot;if&quot;</span>                    <span class="keywordflow">return</span> KEYWORD_IF;</div><div class="line"><span class="stringliteral">&quot;*&quot;</span>                     <span class="keywordflow">return</span> OP_MUL;</div><div class="line"><span class="stringliteral">&quot;=&quot;</span>                     <span class="keywordflow">return</span> OP_ASSIGN;</div><div class="line">[a-zA-Z_][a-zA-Z0-9_]*  <span class="keywordflow">return</span> ASCII_IDENTIFIER;</div><div class="line">[0-9]+                  <span class="keywordflow">return</span> CONST_NUMBER;</div><div class="line">\<span class="stringliteral">&quot;([^\\&quot;</span>]|\\<span class="stringliteral">&quot;)*\&quot;       return CONST_STRING;</span></div><div class="line"><span class="stringliteral">[ \t\r\n]               /* no action: ignore all white space */</span></div><div class="line"><span class="stringliteral">%%</span></div></div><!-- fragment --> </div><p>When the tokenizer matches a pattern, the corresponding action is invoked. The example above returns tokens to the compiler's parser, which repeatedly invokes the tokenizer for more tokens until the tokenizer reaches the end of the input. The tokenizer returns zero (0) when the end of the input is reached.</p>
<p>Lex and Flex have remained relatively stable (inert) tools while the demand has increased for tokenizing Unicode texts encoded in common wide character formats such as UTF-8, UCS/UTF-16, and UTF-32. Also the regular expression syntax in Flex/Lex is limited compared to modern regex syntax. Flex has no support for Unicode patterns, no lazy repetitions, no word boundary anchors, no indentation matching with indent or dedent anchors, and a very limited collection of meta escapes to choose from. To make things even more interesting, it is a challenge to write lexer specifications that avoid the "greedy trap" of POSIX matching.</p>
<p>Flex/Lex scanners use POSIX pattern matching, meaning that the leftmost longest match is returned (among a set of patterns that match the same input). Because POSIX matchers produce the longest match for any given input text, we should be careful when using patterns with "greedy" repetitions (<code>X*</code>, <code>X+</code> etc.) because our pattern may gobble up more input than intended. We end up falling into the "greedy trap".</p>
<p>To illustrate this trap consider matching HTML comments <code>&lt;!−− ... −−&gt;</code> with the pattern <code>&lt;!−−.*−−&gt;</code>. The problem is that the repetition <code>X*</code> is greedy and the <code>.*−−&gt;</code> pattern matches everything until the last <code>−−&gt;</code> while moving over <code>−−&gt;</code> that are between the <code>&lt;!−−</code> and the last <code>−−&gt;</code>.</p>
<dl class="section note"><dt>Note</dt><dd>Dot <code>.</code> normally does not match newline <code>\n</code> in Flex/Lex patterns, unless we use <em>dot-all mode</em> that is sometimes confusingly called "single line mode".</dd></dl>
<p>We can use much more complex patterns such as <code>&lt;!−−([^−]|−[^−]|−−+[^−&gt;])*−*−−&gt;</code> just to match comments in HTML, by ensuring the pattern ends at the first match of a <code>−−&gt;</code> in the input and not at the very last <code>−−&gt;</code> in the input. The POSIX leftmost longest match can be surprisingly effective in rendering our tokenizer into works of ASCII art!</p>
<p>We may claim our intricate pattern trophies as high achievements to the project team, but our team will quickly point out that a regex <code>&lt;!−−.*?−−&gt;</code> suffices to match HTML comments with the <em>lazy repetition</em> <code>X*?</code> construct, also known as a <em>non-greedy repeat</em>. The <code>?</code> is a <em>lazy quantifier</em> that modifies the behavior of the <code>X*?</code> repeat to match only <code>X</code> repeately if the rest of the pattern does not match. Therefore, the regex <code>&lt;!−−.*?−−&gt;</code> matches HTML comments and nothing more.</p>
<p>But Flex/Lex does not permit us to be lazy!</p>
<p>Not surprising, even the Flex manual shows ad-hoc code rather than a pattern to scan over C/C++ source code input to match multiline comments that start with a <code>/*</code> and end with the first occurrence of a <code>*/</code>. The Flex manual recommends:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="stringliteral">&quot;/*&quot;</span>    {  <span class="comment">/* skip multiline comments */</span></div><div class="line">  <span class="keywordtype">int</span> c;</div><div class="line">  <span class="keywordflow">while</span> ((c = <a class="code" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a">yyinput</a>()) != 0)</div><div class="line">  {</div><div class="line">    <span class="keywordflow">if</span> (c == <span class="charliteral">&#39;\n&#39;</span>)</div><div class="line">      ++mylineno;</div><div class="line">    <span class="keywordflow">else</span> <span class="keywordflow">if</span> (c == <span class="charliteral">&#39;*&#39;</span>)</div><div class="line">    {</div><div class="line">      <span class="keywordflow">if</span> ((c = <a class="code" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a">yyinput</a>()) == <span class="charliteral">&#39;/&#39;</span>)</div><div class="line">        <span class="keywordflow">break</span>;</div><div class="line">      unput(c);</div><div class="line">    }</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --> </div><p>Another argument to use this code with Flex is that the internal Flex buffer is limited to 16KB. By contrast, RE/flex buffers are dynamically resized and will never run out of buffer space to accept long matches.</p>
<p>Workarounds such as these are not necessary with RE/flex. The RE/flex scanners use regex libraries with expressive pattern syntax. We can use lazy repetition to write a regex pattern for multiline comments as follows:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="stringliteral">&quot;/*&quot;</span>(.|\n)*?<span class="stringliteral">&quot;*/&quot;</span>  <span class="comment">/* no action: ignore multiline comments */</span></div></div><!-- fragment --> </div><p>Most regex libraries support syntaxes and features that we have come to rely on for pattern matching. A regex with lazy quantifiers can be much easier to read and comprehend compared to a greedy variant. Most regex libraries that support lazy quantifiers run in Perl mode, using backtracking over the input. Scanners use POSIX mode matching, meaning that the leftmost longest match is found. The difference is important as we saw earlier and even more so when we consider the problems with Perl mode matching when specifying patterns to tokenize input, as we will explain next.</p>
<p>Consider the lexer specification example shown earlier. Suppose the input text to tokenize is <code>iflag = 1</code>. In POSIX mode we return <code>ASCII_IDENTIFIER</code> for the name <code>iflag</code>, <code>OP_ASSIGN</code> for <code>=</code>, and <code>NUMBER</code> for <code>1</code>. In Perl mode, we find that <code>iflag</code> matches <code>if</code> and the rest of the name is not consumed, which gives <code>KEYWORD_IF</code> for <code>if</code>, <code>ASCII_IDENTIFIER</code> for <code>lag</code>, <code>OP_ASSIGN</code> for <code>=</code>, and a <code>NUMBER</code> for <code>1</code>. Perl mode matching greedely returns leftmost matches.</p>
<p>Using Perl mode in a scanner requires all overlapping patterns to be defined in a lexer specification such that all longest matching patterns are defined first to ensure longest matches. By contrast, POSIX mode is <em>declarative</em> and allows you to define the patterns in the specification in any order. Perhaps the only ordering constraint on patterns is for patterns that match the same input, such such as matching the keyword <code>if</code> in the example: <code>KEYWORD_IF</code> must be matched before <code>ASCII_IDENTIFIER</code>.</p>
<p>For this reason, RE/flex scanners use a regex library in POSIX mode by default.</p>
<p>In summary:</p>
<ul>
<li>RE/flex specifications and use are compatible with Lex/Flex and Bison/Yacc.</li>
<li>The RE/flex scanner generator accepts specifications that are compatible with <a href="dinosaur.compilertools.net/#flex">Flex</a>, with additional options to select a matcher engine and to specify names and options for C++ class generation.</li>
<li>The RE/flex scanner generator option <code>−−bison</code> generates a scanner compatible with <a href="dinosaur.compilertools.net/#bison">Bison</a>. RE/flex also offers options to integrate Bison bridge, Bison locations, Bison complete, and reentrant parsers.</li>
<li>The regular expression syntax in Flex and Lex specifications is restricted to POSIX ERE. By contrast, the RE/flex specification syntax is regex-centric by design and offers a much richer pattern syntax, including lazy quantifiers.</li>
<li>A matcher engine for a lexer class has a common interface API declared by the abstract base matcher class template.</li>
<li>RE/flex scanners are not implemented as a set of global functions and tables. RE/flex scanners are instances of generated lexer classes, which ensures that scanners are thread-safe (by contrast, Flex and Flex++ are not thread-safe). A lexer class is derived from an abstract base lexer class template and it is instantiated with a regex matcher engine class that is provided as a template parameter.</li>
<li>Lex and Flex scanners are restricted to ASCII or 8-bit character sets without adequate support for Unicode. RE/flex specifications are extended to support Unicode such that the RE/flex scanner generator produces scanners that handle UTF-8/16/32 input files automatically.</li>
<li>Input to matcher engines and scanners is implemented as a class that supports streaming sources of input and automatically decodes UTF-encoded files.</li>
<li>RE/flex scanners can be stand-alone applications, similar to Flex scanners.</li>
</ul>
<p><a class="el" href="index.html#reflex">The RE/flex scanner generator </a> section has more details on the RE/flex scanner generator tool.</p>
<p>In the next part of this manual, we will take a quick look at the RE/flex regex API that can be used as a stand-alone library for matching, searching, scanning and splitting input from strings, files and streams in regular C++ applications (i.e. applications that are not necessarily tokenizers for compilers).</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="intro2"></a>
And a flexible regex library                                          </h2>
<p>The RE/flex regex pattern matching classes include two classes for Boost.Regex, two classes for PCRE2, two classes for C++11 std::regex, and a RE/flex class:</p>
<table class="doxtable">
<tr>
<th>Engine </th><th>Header file to include </th><th>C++ matcher classes  </th></tr>
<tr>
<td>RE/flex regex </td><td><code><a class="el" href="matcher_8h.html" title="RE/flex matcher engine. ">reflex/matcher.h</a></code> </td><td><code>Matcher</code> </td></tr>
<tr>
<td>Boost.Regex </td><td><code><a class="el" href="boostmatcher_8h.html" title="Boost::regex-based matcher engines for pattern matching. ">reflex/boostmatcher.h</a></code> </td><td><code>BoostMatcher</code>, <code>BoostPosixMatcher</code> </td></tr>
<tr>
<td>PCRE2 </td><td><code><a class="el" href="pcre2matcher_8h.html" title="PCRE2-based matcher engines for pattern matching. ">reflex/pcre2matcher.h</a></code> </td><td><code>PCRE2Matcher</code>, <code>PCRE2UTFMatcher</code> </td></tr>
<tr>
<td>std::regex </td><td><code><a class="el" href="stdmatcher_8h.html" title="C++11 std::regex-based matcher engines for pattern matching. ">reflex/stdmatcher.h</a></code> </td><td><code>StdMatcher</code>, <code>StdPosixMatcher</code> </td></tr>
</table>
<p>The RE/flex <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> class compiles regex patterns to efficient finite state machines (FSMs) when instantiated. These deterministic automata speed up matching considerably, at the cost of the initial FSM construction (see further below for hints on how to avoid this run time overhead). RE/flex matchers only support POSIX mode matching, see <a class="el" href="index.html#reflex-posix-perl">POSIX versus Perl matching </a>.</p>
<p>The <code><a class="el" href="classreflex_1_1_boost_matcher.html" title="Boost matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::BoostMatcher</a></code> and <code><a class="el" href="classreflex_1_1_boost_posix_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost POSIX regex matching. ">reflex::BoostPosixMatcher</a></code> classes are for Perl mode and POSIX mode matching, respectively.</p>
<p>The <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html" title="PCRE2 JIT-optimized matcher engine class implements reflex::PatternMatcher pattern matching interface...">reflex::PCRE2Matcher</a></code> and <code><a class="el" href="classreflex_1_1_p_c_r_e2_u_t_f_matcher.html" title="PCRE2 JIT-optimized native PCRE2_UTF+PCRE2_UCP matcher engine class, extends PCRE2Matcher. ">reflex::PCRE2UTFMatcher</a></code> classes are for Perl mode matching only, where the latter uses native PCRE2 Unicode matching with <code>PCRE2_UTF+PCRE2_UCP</code>. The PCRE2 matchers use JIT optimizations to speed up matching, which comes at a cost of extra processing when the matcher is instantiated. The benefit outweighs the cost when many matches are processed.</p>
<p>C++11 std::regex supports ECMAScript and AWK POSIX syntax with the <code>StdMatcher</code> and <code><a class="el" href="classreflex_1_1_std_posix_matcher.html" title="std matcher engine class, extends reflex::StdMatcher for POSIX ERE std::regex::awk syntax and regex m...">reflex::StdPosixMatcher</a></code> classes respectively. The std::regex syntax is therefore a lot more limited compared to Boost.Regex, PCRE2, and RE/flex.</p>
<p>The RE/flex regex common interface API is implemented in an abstract base class template <code><a class="el" href="classreflex_1_1_abstract_matcher.html" title="The abstract matcher base class template defines an interface for all pattern matcher engines...">reflex::AbstractMatcher</a></code> from which regex matchers are derived. This regex API offers a common interface that is used in the generated scanner. You can also use this API in your C++ application for pattern matching.</p>
<p>The RE/flex abstract matcher offers four operations for matching with the regex engines that are derived from this base abstract class:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>matches()</code> </td><td>returns nonzero if the input from begin to end matches </td></tr>
<tr>
<td><code>find()</code> </td><td>search input and return nonzero if a match was found </td></tr>
<tr>
<td><code>scan()</code> </td><td>return nonzero if input at current position matches partially </td></tr>
<tr>
<td><code>split()</code> </td><td>return nonzero for a split of the input at the next match </td></tr>
</table>
<p>These methods return a nonzero value for a match, meaning the <code>size_t accept()</code> value, and are repeatable, where the last three return additional matches.</p>
<p>For example, to check if a string is a valid date using Boost.Regex:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span> <span class="comment">// reflex::BoostMatcher, reflex::Input, boost::regex</span></div><div class="line"></div><div class="line"><span class="comment">// use a BoostMatcher to check if the birthdate string is a valid date</span></div><div class="line"><span class="keywordflow">if</span> (<a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a>(<span class="stringliteral">&quot;\\d{4}-\\d{2}-\\d{2}&quot;</span>, birthdate).matches())</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Valid date!&quot;</span> &lt;&lt; std::endl;</div></div><!-- fragment --><p>We can perform exactly the same check with PCRE2 instead of Boost.Regex. However, the JIT-optimized PCRE2 matcher is better suited when many matches are processed, not just one as shown here:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// use a PCRE2Matcher to check if the birthdate string is a valid date</span></div><div class="line"><span class="keywordflow">if</span> (<a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a>(<span class="stringliteral">&quot;\\d{4}-\\d{2}-\\d{2}&quot;</span>, birthdate).matches())</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Valid date!&quot;</span> &lt;&lt; std::endl;</div></div><!-- fragment --><p>Swapping regex libraries is simple. Sometimes we may need a regex converter when a regex feature is used that the regex library does not support.</p>
<p>To search a string for all words matching the pattern <code>\w+</code>:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// use a PCRE2Matcher to search for words in a sentence</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">Found How
Found now
Found brown
Found cow
</pre><p>If we want to match Unicode words, <code>\w+</code> should be converted to a Unicode pattern, here we convert the pattern for matching with Boost.Regex:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span> <span class="comment">// reflex::BoostMatcher, reflex::Input, boost::regex</span></div><div class="line"></div><div class="line"><span class="comment">// convert \w+ for Unicode matching</span></div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> std::string pattern = <a class="code" href="classreflex_1_1_boost_matcher.html#ad1827615da2d51f4896e9c802b6463fc">reflex::BoostMatcher::convert</a>(<span class="stringliteral">&quot;\\w+&quot;</span>, <a class="code" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39">reflex::convert_flag::unicode</a>);</div><div class="line"></div><div class="line"><span class="comment">// use a BoostMatcher to search for words in a sentence</span></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher(pattern, <span class="stringliteral">&quot;Höw nöw bröwn cöw.&quot;</span>);</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>Conversion to Unicode patterns is necessary for all matchers except <code><a class="el" href="classreflex_1_1_p_c_r_e2_u_t_f_matcher.html" title="PCRE2 JIT-optimized native PCRE2_UTF+PCRE2_UCP matcher engine class, extends PCRE2Matcher. ">reflex::PCRE2UTFMatcher</a></code>, since matchers operate in non-Unicode mode by default to match bytes, not wide characters. We will come back again to converters later.</p>
<p>When executed this code prints: </p><pre class="fragment">Found Höw
Found nöw
Found bröwn
Found cöw
</pre><p>The same code and results are produced with <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html" title="PCRE2 JIT-optimized matcher engine class implements reflex::PatternMatcher pattern matching interface...">reflex::PCRE2Matcher</a></code> defined in <code><a class="el" href="pcre2matcher_8h.html" title="PCRE2-based matcher engines for pattern matching. ">reflex/pcre2matcher.h</a></code>. For the following examples we will use Boost.Regex or PCRE2, which may be used interchangeably.</p>
<p>The <code>scan</code> method is similar to the <code>find</code> method, but <code>scan</code> matches only from the current position in the input. It fails when no partial match was possible at the current position. Repeately scanning an input source means that matches must be continuous, otherwise <code>scan</code> returns zero (no match).</p>
<p>The <code>split</code> method is roughly the inverse of the <code>find</code> method and returns text located between matches. For example using non-word matching <code>\W+</code>:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// use a PCRE2Matcher to search for words in a sentence</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\W+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordflow">while</span> (matcher.split() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">Found How
Found now
Found brown
Found cow
Found
</pre><p>Note that split also returns the (possibly empty) remaining text after the last match, as you can see in the output above: the last split with <code>\W+</code> returns an empty string, which is the remaining input after the period in the sentence.</p>
<p>The <code>find()</code>, <code>scan()</code> and <code>split()</code> methods return a nonzero *"accept"* value, which corresponds to the regex group captured, or the methods return zero if no match was found. The methods return 1 for a match when no groups are used. The <code>split()</code> method has a special case. It returns the value <code><a class="el" href="structreflex_1_1_abstract_matcher_1_1_const.html#a7c42ab60928bcc59f420876e741aae0b" title="accept() returns &quot;empty&quot; last split at end of input ">reflex::PCRE2Matcher::Const::EMPTY</a></code> (and so does any other matcher) when a match was made at the end of the input and an empty string was split, as is the case of the last <code>split()</code> match in the example above.</p>
<p>Another example:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// use a PCRE2Matcher to tokenize a string</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;(\\w+)|(\\s+)|(.)&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordtype">size_t</span> accept;</div><div class="line"><span class="keywordflow">while</span> ((accept = matcher.scan()) != 0)</div><div class="line">{</div><div class="line">  <span class="keywordflow">switch</span> (accept)</div><div class="line">  {</div><div class="line">    <span class="keywordflow">case</span> 1: std::cout &lt;&lt; <span class="stringliteral">&quot;word\n&quot;</span>;  <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> 2: std::cout &lt;&lt; <span class="stringliteral">&quot;space\n&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> 3: std::cout &lt;&lt; <span class="stringliteral">&quot;other\n&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">word
space
word
space
word
space
word
other
</pre><p>The regex engines currently available as classes in the <code>reflex</code> namespace are:</p>
<table class="doxtable">
<tr>
<th>Class </th><th>Mode </th><th>Engine </th><th>Performance  </th></tr>
<tr>
<td><code>Matcher</code> </td><td>POSIX </td><td>RE/flex lib </td><td>deterministic finite automaton, no backtracking </td></tr>
<tr>
<td><code>BoostMatcher</code> </td><td>Perl </td><td>Boost.Regex </td><td>backtracking </td></tr>
<tr>
<td><code>BoostPerlMatcher</code> </td><td>Perl </td><td>Boost.Regex </td><td>backtracking </td></tr>
<tr>
<td><code>BoostPosixMatcher</code> </td><td>POSIX </td><td>Boost.Regex </td><td>backtracking </td></tr>
<tr>
<td><code>PCRE2Matcher</code> </td><td>Perl </td><td>PCRE2 </td><td>JIT-optimized backtracking </td></tr>
<tr>
<td><code>PCRE2UTFMatcher</code> </td><td>Perl </td><td>PCRE2 UTF+UPC </td><td>JIT-optimized backtracking </td></tr>
<tr>
<td><code>StdMatcher</code> </td><td>ECMA </td><td>std::regex </td><td>backtracking </td></tr>
<tr>
<td><code>StdEcmaMatcher</code> </td><td>ECMA </td><td>std::regex </td><td>backtracking </td></tr>
<tr>
<td><code>StdPosixMatcher</code> </td><td>POSIX </td><td>std::regex </td><td>backtracking </td></tr>
</table>
<p>The RE/flex regex engine uses a deterministic finite state machine (FSM) to get the best performance when matching. However, constructing a FSM adds overhead. This matcher is better suitable for searching long texts. The FSM construction overhead can be eliminated by pre-converting the regex to C++ code tables ahead of time as we will see shortly.</p>
<p>The Boost.Regex engines normally use Perl mode matching. We added a POSIX mode Boost.Regex engine class for the RE/flex scanner generator. Scanners typically use POSIX mode matching. See <a class="el" href="index.html#reflex-posix-perl">POSIX versus Perl matching </a> for more information.</p>
<p>The Boost.Regex engines are all initialized with <code>match_not_dot_newline</code>, which disables dotall matching as the default setting. Dotall can be re-enabled with the <code>(?s)</code> regex mode modifier. This is done for compatibility with scanners.</p>
<p>The PCRE2 engines use Perl mode matching. PCRE2 also offers POSIX mode matching with <code>pcre2_dfa_match()</code>. However, group captures are not supported in this mode. Therefore, no PCRE2 POSIX mode class is included as a choice. JIT optimizations speed up matching. However, this comes at a cost of extra processing when the PCRE2 matcher class is instantiated.</p>
<p>A matcher may be applied to strings and wide strings, such as <code>std::string</code> and <code>std::wstring</code>, <code>char*</code> and <code>wchar_t*</code>. Wide strings are converted to UTF-8 to enable matching with regular expressions that contain Unicode patterns.</p>
<p>To match Unicode patterns with regex library engines that are 8-bit based or do not support Unicode, we want to convert your regex string first before we use it with a regex matcher engine:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span> <span class="comment">// reflex::Matcher, reflex::Input, reflex::Pattern</span></div><div class="line"></div><div class="line"><span class="comment">// convert a regex with Unicode character classes to create a pattern FSM:</span></div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> std::string regex = <a class="code" href="classreflex_1_1_matcher.html#a8862686f6d5680d4d0e5af146c4aed69">reflex::Matcher::convert</a>(<span class="stringliteral">&quot;[\\p{Greek}\\p{Zs}\\pP]+&quot;</span>, <a class="code" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39">reflex::convert_flag::unicode</a>);</div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> pattern(regex);</div><div class="line"></div><div class="line"><span class="comment">// use a Matcher to check if sentence is in Greek:</span></div><div class="line"><span class="keywordflow">if</span> (<a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a>(pattern, sentence).matches())</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;This is Greek&quot;</span> &lt;&lt; std::endl;</div></div><!-- fragment --><p>This converts the Unicode character classes to UTF-8 for matching with an 8-bit regex engine. The <code>convert</code> static method differs per matcher class. An error <code><a class="el" href="classreflex_1_1regex__error.html" title="Regex syntax error exceptions. ">reflex::regex_error</a></code> is thrown as an exception if conversion was not possible, which is unlikely, or if the regex is syntactically incorrect.</p>
<p>Conversion is fast (it runs in linear time in the size of the regex), but it is not without some overhead. We should make the converted regex patterns <code>static</code> whenever possible, as shown above, to eliminate the cost of repeated conversions and pattern constructions.</p>
<p>A <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> object is immutable (it stores a constant table) and may be shared among threads.</p>
<p>Use <code>convert</code> with option <code><a class="el" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39" title="convert . (dot), \s, \w, \l, \u, \S, \W, \L, \U to Unicode ">reflex::convert_flag::unicode</a></code> to change the meaning of <code>.</code> (dot), <code>\w</code>, <code>\s</code>, <code>\l</code>, <code>\u</code>, <code>\W</code>, <code>\S</code>, <code>\L</code>, <code>\U</code> character classes.</p>
<p>File contents are streamed into the matcher using partial matching algorithms and matching happens immediately. This means that the input does not need to be loaded as a whole into memory. This supports interactive matching, i.e. matching the input from a console:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// use a PCRE2Matcher to search and display words from console input</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, std::cin);</div><div class="line">matcher.interactive();</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>We can also pattern match text from <code>FILE</code> descriptors. The additional benefit of using <code>FILE</code> descriptors is the automatic decoding of UTF-16/32 input to UTF-8 by the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class that manages input sources and their state.</p>
<p>For example, pattern matching the content of <em><code>"cows.txt"</code></em> that may use UTF-8, 16, or 32 encodings:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// use a PCRE2Matcher to search and display words from a FILE</span></div><div class="line">FILE *fd = fopen(<span class="stringliteral">&quot;cows.txt&quot;</span>, <span class="stringliteral">&quot;r&quot;</span>);</div><div class="line"><span class="keywordflow">if</span> (fd == NULL)</div><div class="line">  exit(EXIT_FAILURE);</div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, fd);</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>The <code>find</code>, <code>scan</code>, and <code>split</code> methods are also implemented as input iterators that apply filtering tokenization, and splitting:</p>
<table class="doxtable">
<tr>
<th>Iterator range </th><th>Acts as a </th><th>Iterates over  </th></tr>
<tr>
<td><code>find.begin()</code>...<code>find.end()</code> </td><td>filter </td><td>all matches </td></tr>
<tr>
<td><code>scan.begin()</code>...<code>scan.end()</code> </td><td>tokenizer </td><td>continuous matches </td></tr>
<tr>
<td><code>split.begin()</code>...<code>split.end()</code> </td><td>splitter </td><td>text between matches </td></tr>
</table>
<p>The type <code><a class="el" href="classreflex_1_1_abstract_matcher_1_1_operation.html" title="AbstractMatcher::Operation functor to match input to a pattern, also provides a (const) AbstractMatch...">reflex::AbstractMatcher::Operation</a></code> is a functor that defines <code>find</code>, <code>scan</code>, and <code>split</code>. The functor operation returns true upon success. The use of an iterator is simply supported by invoking <code>begin()</code> and <code>end()</code> methods of the functor, which return <code><a class="el" href="classreflex_1_1_abstract_matcher.html#a7237c1fb8ab2a2283117739e6516c31f" title="std::input_iterator for scanning, searching, and splitting input character sequences ...">reflex::AbstractMatcher::iterator</a></code>. Likewise, there are also <code>cbegin()</code> and <code>cend()</code> methods that return a <code>const_iterator</code>.</p>
<p>We can use these RE/flex iterators in C++ for many tasks, including to populate containers by stuffing the iterator's text matches into it:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"><span class="preprocessor">#include &lt;vector&gt;</span>                <span class="comment">// std::vector</span></div><div class="line"></div><div class="line"><span class="comment">// use a PCRE2Matcher to convert words of a sentence into a string vector</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line">std::vector&lt;std::string&gt; words(matcher.find.begin(), matcher.find.end());</div></div><!-- fragment --><p>As a result, the <code>words</code> vector contains "How", "now", "brown", "cow".</p>
<p>Casting a matcher object to <code>std::string</code> is the same as converting <code>text()</code> to a string with <code>std::string(text(), size())</code>, which in the example above is done to construct the <code>words</code> vector. Casting a matcher object to <code>std::wstring</code> is similar, but also converts the UTF-8 <code>text()</code> match to a wide string.</p>
<p>RE/flex iterators are useful in C++11 range-based loops. For example:</p>
<div class="fragment"><div class="line"><span class="comment">// Requires C++11, compile with: cc -std=c++11</span></div><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="stdmatcher_8h.html">reflex/stdmatcher.h</a>&gt;</span> <span class="comment">// reflex::StdMatcher, reflex::Input, std::regex</span></div><div class="line"></div><div class="line"><span class="comment">// use a StdMatcher to search for words in a sentence using an iterator</span></div><div class="line"><a class="code" href="classreflex_1_1_std_matcher.html">reflex::StdMatcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordflow">for</span> (<span class="keyword">auto</span>&amp; match : matcher.find)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; match.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">Found How
Found now
Found brown
Found cow
</pre><p>And RE/flex iterators are also useful with STL algorithms and lambdas, for example to compute a histogram of word frequencies:</p>
<div class="fragment"><div class="line"><span class="comment">// Requires C++11, compile with: cc -std=c++11</span></div><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="stdmatcher_8h.html">reflex/stdmatcher.h</a>&gt;</span> <span class="comment">// reflex::StdMatcher, reflex::Input, std::regex</span></div><div class="line"><span class="preprocessor">#include &lt;algorithm&gt;</span>           <span class="comment">// std::for_each</span></div><div class="line"></div><div class="line"><span class="comment">// use a StdMatcher to create a frequency histogram of group captures</span></div><div class="line"><a class="code" href="classreflex_1_1_std_matcher.html">reflex::StdMatcher</a> matcher(<span class="stringliteral">&quot;(now)|(cow)|(ow)&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordtype">size_t</span> freq[4] = { 0, 0, 0, 0 };</div><div class="line">std::for_each(matcher.find.begin(), matcher.find.end(), [&amp;](<span class="keywordtype">size_t</span> n){ ++freq[n]; });</div></div><!-- fragment --><p>As a result, the <code>freq</code> array contains 0, 1, 1, and 2.</p>
<p>Casting the matcher object to a <code>size_t</code> returns the group capture index, which is used in the example shown above. We also us it in the example below that is capturing all regex pattern groupings into a vector:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"><span class="preprocessor">#include &lt;vector&gt;</span>                <span class="comment">// std::vector</span></div><div class="line"></div><div class="line"><span class="comment">// use a PCRE2Matcher to convert captured groups into a numeric vector</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;(now)|(cow)|(ow)&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line">std::vector&lt;size_t&gt; captures(matcher.find.begin(), matcher.find.end());</div></div><!-- fragment --><p>As a result, the vector contains the group captures 3, 1, 3, and 2.</p>
<p>Casting the matcher object to <code>size_t</code> is the same as invoking <code>accept()</code>.</p>
<p>This method and other methods may be used to obtain the details of a match:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>accept()</code> </td><td>returns group capture index (or zero if not captured/matched) </td></tr>
<tr>
<td><code>text()</code> </td><td>returns <code>const char*</code> to 0-terminated match (ends in <code>\0</code>) </td></tr>
<tr>
<td><code>str()</code> </td><td>returns <code>std::string</code> text match (preserves <code>\0</code>s) </td></tr>
<tr>
<td><code>wstr()</code> </td><td>returns <code>std::wstring</code> wide text match (converted from UTF-8) </td></tr>
<tr>
<td><code>chr()</code> </td><td>returns first 8-bit character of the text match (<code>str()[0]</code>) </td></tr>
<tr>
<td><code>wchr()</code> </td><td>returns first wide character of the text match (<code>wstr()[0]</code>) </td></tr>
<tr>
<td><code>pair()</code> </td><td>returns <code>std::pair&lt;size_t,std::string&gt;(accept(),str())</code> </td></tr>
<tr>
<td><code>wpair()</code> </td><td>returns <code>std::pair&lt;size_t,std::wstring&gt;(accept(),wstr())</code> </td></tr>
<tr>
<td><code>size()</code> </td><td>returns the length of the text match in bytes </td></tr>
<tr>
<td><code>wsize()</code> </td><td>returns the length of the match in number of wide characters </td></tr>
<tr>
<td><code>lines()</code> </td><td>returns the number of lines in the text match (&gt;=1) </td></tr>
<tr>
<td><code>columns()</code> </td><td>returns the number of columns of the text match (&gt;=0) </td></tr>
<tr>
<td><code>begin()</code> </td><td>returns <code>const char*</code> to non-0-terminated text match begin </td></tr>
<tr>
<td><code>end()</code> </td><td>returns <code>const char*</code> to non-0-terminated text match end </td></tr>
<tr>
<td><code>rest()</code> </td><td>returns <code>const char*</code> to 0-terminated rest of input </td></tr>
<tr>
<td><code>span()</code> </td><td>returns <code>const char*</code> to 0-terminated match enlarged to span the line </td></tr>
<tr>
<td><code>line()</code> </td><td>returns <code>std::string</code> line with the matched text as a substring </td></tr>
<tr>
<td><code>wline()</code> </td><td>returns <code>std::wstring</code> line with the matched text as a substring </td></tr>
<tr>
<td><code>more()</code> </td><td>tells the matcher to append the next match (when using <code>scan()</code>) </td></tr>
<tr>
<td><code>less(n)</code> </td><td>cuts <code>text()</code> to <code>n</code> bytes and repositions the matcher </td></tr>
<tr>
<td><code>lineno()</code> </td><td>returns line number of the match, starting at line 1 </td></tr>
<tr>
<td><code>columno()</code> </td><td>returns column number of the match in characters, starting at 0 </td></tr>
<tr>
<td><code>lineno_end()</code> </td><td>returns ending line number of the match, starting at line 1 </td></tr>
<tr>
<td><code>columno_end()</code> </td><td>returns ending column number of the match, starting at 0 </td></tr>
<tr>
<td><code>border()</code> </td><td>returns the byte offset from the start of the line of the match </td></tr>
<tr>
<td><code>first()</code> </td><td>returns input position of the first character of the match </td></tr>
<tr>
<td><code>last()</code> </td><td>returns input position + 1 of the last character of the match </td></tr>
<tr>
<td><code>at_bol()</code> </td><td>true if matcher reached the begin of a new line <code>\n</code> </td></tr>
<tr>
<td><code>at_bob()</code> </td><td>true if matcher is at the begin of input and no input consumed </td></tr>
<tr>
<td><code>at_end()</code> </td><td>true if matcher is at the end of input </td></tr>
<tr>
<td><code>[0]</code> </td><td>operator returns <code>std::pair&lt;const char*,size_t&gt;(begin(),size())</code> </td></tr>
<tr>
<td><code>[n]</code> </td><td>operator returns n'th capture <code>std::pair&lt;const char*,size_t&gt;</code> </td></tr>
</table>
<p>For a detailed explanation of these methods, see <a class="el" href="index.html#regex-methods-props">Properties of a match</a>.</p>
<p>The <code>operator[n]</code> takes the group number <code>n</code> and returns the n'th group capture match as a pair with a <code>const char*</code> pointer to the group-matching text and the size of the matched text in bytes. Because the pointer points to a string that is not 0-terminated, use the size to determine the matching part.</p>
<p>The pointer is <code>NULL</code> when the group capture has no match.</p>
<p>For example:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// a PCRE2Matcher to capture name and number:</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;(\\w+)\\s+(\\d+)&quot;</span>);</div><div class="line"></div><div class="line"><span class="comment">// use the matcher on a string:</span></div><div class="line"><span class="keywordflow">if</span> (matcher.input(<span class="stringliteral">&quot;cow 123&quot;</span>).matches())</div><div class="line">  std::cout &lt;&lt;</div><div class="line">    <span class="stringliteral">&quot;name: &quot;</span> &lt;&lt; std::string(matcher[1].first, matcher[1].second) &lt;&lt;</div><div class="line">    <span class="stringliteral">&quot;, number: &quot;</span> &lt;&lt; std::string(matcher[2].first, matcher[2].second) &lt;&lt;</div><div class="line">    std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">name: cow, number: 123
</pre><dl class="section warning"><dt>Warning</dt><dd>The <code>text()</code> method returns the match by pointing to the <code>const char*</code> string that is stored in an internal buffer. This pointer <em>should not be used</em> after matching continues and when the matcher object is deallocated. To retain the <code>text()</code> value use the <code>str()</code> method that returns a copy of <code>text()</code>.</dd>
<dd>
The <code>operator[]</code> method returns a pair with the match info of the n'th group, which is a non-0-terminated <code>const char*</code> pointer (or NULL) and its size in bytes of the captured match. The string <em>should not be used</em> after matching continues.</dd></dl>
<dl class="section note"><dt>Note</dt><dd>When using the <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> class, the <code>accept()</code> method returns the accepted pattern among the alternations in the regex that are specified only at the top level in the regex. For example, the regex <code>"(a(b)c)|([A-Z])"</code> has two groups, because only the outer top-level groups are recognized. Because groups are specified at the top level only, the grouping parenthesis are optional. We can simplify the regex to <code>"a(b)c|[A-Z]"</code> and still capture the two patterns.</dd></dl>
<p>The following methods may be used to manipulate the input stream directly:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>input()</code> </td><td>returns next 8-bit char from the input, matcher then skips it </td></tr>
<tr>
<td><code>winput()</code> </td><td>returns next wide character from the input, matcher skips it </td></tr>
<tr>
<td><code>unput(c)</code> </td><td>put 8-bit char <code>c</code> back unto the stream, matcher then takes it </td></tr>
<tr>
<td><code>wunput(c)</code></td><td>put (wide) char <code>c</code> back unto the stream, matcher then takes it </td></tr>
<tr>
<td><code>peek()</code> </td><td>returns next 8-bit char from the input without consuming it </td></tr>
<tr>
<td><code>skip(c)</code> </td><td>skip input until character <code>c</code> (<code>char</code> or <code>wchar_t</code>) is consumed </td></tr>
<tr>
<td><code>skip(s)</code> </td><td>skip input until UTF-8 string <code>s</code> is consumed </td></tr>
<tr>
<td><code>rest()</code> </td><td>returns the remaining input as a 0-terminated <code>char*</code> string </td></tr>
</table>
<p>The <code>input()</code>, <code>winput()</code>, and <code>peek()</code> methods return a non-negative character code and EOF (-1) when the end of input is reached.</p>
<p>To initialize a matcher for interactive use, to assign a new input source or to change its pattern, use the following methods:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>input(i)</code> </td><td>set input to <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> (string, stream, or <code>FILE*</code>) </td></tr>
<tr>
<td><code>pattern(p)</code> </td><td>set pattern <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code>, <code>boost::regex</code>, or a string <code>p</code> </td></tr>
<tr>
<td><code>has_pattern()</code> </td><td>true if the matcher has a pattern assigned to it </td></tr>
<tr>
<td><code>own_pattern()</code> </td><td>true if the matcher has a pattern to manage and delete </td></tr>
<tr>
<td><code>pattern()</code> </td><td>a reference to the pattern object </td></tr>
<tr>
<td><code>interactive()</code> </td><td>sets buffer size to 1 for console-based (TTY) input </td></tr>
<tr>
<td><code>buffer()</code> </td><td>buffer all input at once, returns true if successful </td></tr>
<tr>
<td><code>buffer(n)</code> </td><td>set the initial buffer size to <code>n</code> bytes to buffer input </td></tr>
<tr>
<td><code>buffer(b, n)</code> </td><td>read <code>n</code> bytes at address <code>b</code> containing a string of <code>n</code>-1 bytes (zero copy) </td></tr>
<tr>
<td><code>flush()</code> </td><td>flush the remaining input from the internal buffer </td></tr>
<tr>
<td><code>reset()</code> </td><td>resets the matcher, restarting it from the remaining input </td></tr>
<tr>
<td><code>reset(o)</code> </td><td>resets the matcher with new options string <code>o</code> ("A?N?T?") </td></tr>
</table>
<p>A <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object represents the source of input for a matcher, which is either a file <code>FILE*</code>, or a string (with UTF-8 character data) of <code>const char*</code> or <code>std::string</code> type, or a stream pointer <code>std::istream*</code>. The <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object is implicitly constructed from one of these input sources, for example:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// set the input source to a string (or a stream or a FILE*)</span></div><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> source = <span class="stringliteral">&quot;How now brown cow.&quot;</span>;</div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, source);</div><div class="line"></div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div><div class="line"></div><div class="line"><span class="comment">// use the same matcher with a new source (an Input object):</span></div><div class="line">std::ifstream ifs(<span class="stringliteral">&quot;cows.txt&quot;</span>, std::ios::in | std::ios::binary);</div><div class="line">source = ifs;           <span class="comment">// Input source is reassignable</span></div><div class="line">matcher.input(source);  <span class="comment">// can use ifs as parameter also</span></div><div class="line"></div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div><div class="line"></div><div class="line">ifs.close();</div></div><!-- fragment --><p>The entire input is buffered in a matcher with <code>buffer()</code>, or is read piecemeal with <code>buffer(n)</code>, or is read interactively with <code>interactive()</code>. These methods should be used after setting the input source. Reading a stream with buffering all input data at once is done with the <code>&gt;&gt;</code> operator as a shortcut:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// read and buffer cows.txt file</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;&lt;cow&gt;&quot;</span>);</div><div class="line">std::ifstream cows(<span class="stringliteral">&quot;cows.txt&quot;</span>, std::ios::in | std::ios::binary);</div><div class="line">cows &gt;&gt; matcher;     <span class="comment">// same as matcher.input(cows).buffer();</span></div><div class="line">cows.close();        <span class="comment">// can already close now because stream content is stored</span></div><div class="line"></div><div class="line"><span class="comment">// count number of &#39;cow&#39; words:</span></div><div class="line">std::out &lt;&lt; std::distance(matcher.find.begin(), matcher.find.end()) &lt;&lt; <span class="stringliteral">&quot; &#39;cow&#39; in cows.txt\n&quot;</span>;</div></div><!-- fragment --><p>Zero-copy overhead is achieved by specifying <code>buffer(b, n)</code> to read <code>n</code>-1 bytes at address <code>b</code> for in-place matching, where bytes <code>b[0...n]</code> are possibly modified by the matcher:</p>
<div class="fragment"><div class="line"><span class="comment">// read a 0-terminated buffer in place, buffer content is changed!!</span></div><div class="line"><span class="keywordtype">char</span> *base = ...;  <span class="comment">// points to 0-terminated buffer</span></div><div class="line"><span class="keywordtype">size_t</span> size = ...; <span class="comment">// length of the buffer including final \0 byte</span></div><div class="line">matcher.buffer(base, size);</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div></div><!-- fragment --><dl class="section warning"><dt>Warning</dt><dd><code>buffer(b, n)</code> reads <code>n</code>-1 bytes at address <code>b</code>. The length <code>n</code> should include the final zero byte at the end of the string.</dd></dl>
<dl class="section note"><dt>Note</dt><dd>In fact, the specified string may have any final byte value. The final byte of the string will be set to zero when <code>text()</code> or <code>rest()</code> are used. Only <code>unput(c)</code>, <code>wunput()</code>, <code>text()</code>, <code>rest()</code>, and <code>span()</code> modify the buffer contents, because these functions require an extra byte at the end of the buffer to make the strings returned by these methods 0-terminated. This means that we can specify read-only memory of <code>n</code> bytes located at address <code>b</code> by using <code>buffer(b, n+1)</code> safely as long as we do not use <code>unput()</code>, <code>unput()</code>, <code>text()</code>, <code>rest()</code>, and <code>span()</code>, for example to search read-only mmap(2) <code>PROT_READ</code> memory.</dd></dl>
<p>So far we explained how to use <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html" title="PCRE2 JIT-optimized matcher engine class implements reflex::PatternMatcher pattern matching interface...">reflex::PCRE2Matcher</a></code> and <code><a class="el" href="classreflex_1_1_boost_matcher.html" title="Boost matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::BoostMatcher</a></code> for pattern matching. We can also use the RE/flex <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> class for pattern matching. The API is exactly the same. The <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> class uses <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code>, which internally represents an efficient finite state machine that is compiled from a regex. These state machines are used for fast matching.</p>
<p>The construction of deterministic finite state machines (FSMs) is optimized but can take some time and therefore adds overhead before matching can start. This FSM construction should not be executed repeatedly if it can be avoided. So we recommend to construct static pattern objects to create the FSMs only once:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span> <span class="comment">// reflex::Matcher, reflex::Pattern, reflex::Input</span></div><div class="line"></div><div class="line"><span class="comment">// statically allocate and construct a pattern, i.e. once and for all</span></div><div class="line"><span class="keyword">static</span> <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> word_pattern(<span class="stringliteral">&quot;\\w+&quot;</span>);</div><div class="line"></div><div class="line"><span class="comment">// use the RE/flex POSIX matcher to search for words in a string sentence</span></div><div class="line"><a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a> matcher(word_pattern, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>A <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> object is immutable (it stores a constant table) and may be shared among threads.</p>
<p>The RE/flex matcher only supports POSIX mode matching and does not support Perl mode matching. See <a class="el" href="index.html#reflex-posix-perl">POSIX versus Perl matching </a> for more information.</p>
<p>The RE/flex <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> class has several options that control the regex. Options and modes for the regex are set as a string, for example:</p>
<div class="fragment"><div class="line"><span class="keyword">static</span> <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> word_pattern(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;f=graph.gv;f=machine.cpp&quot;</span>);</div></div><!-- fragment --><p>The <code>f=graph.gv</code> option emits a Graphviz <em><code>.gv</code></em> file that can be visually rendered with the open source <a href="http://www.graphviz.org">Graphviz dot tool</a> by converting the deterministic finite state machine (FSM) to PDF, PNG, or other formats:</p>
<div align="center">
<img src="dot_inline_dotgraph_1.png" alt="dot_inline_dotgraph_1.png" border="0" usemap="#dot_inline_dotgraph_1.map"/>
<map name="dot_inline_dotgraph_1.map" id="dot_inline_dotgraph_1.map"></map>
</div>
<p>The <code>f=machine.cpp</code> option emits opcode tables for the finite state machine, which in this case is the following table of 11 code words:</p>
<div class="fragment"><div class="line">REFLEX_CODE_DECL reflex_code_FSM[11] =</div><div class="line">{</div><div class="line">  0x617A0005, <span class="comment">// 0: GOTO 5 ON a-z</span></div><div class="line">  0x5F5F0005, <span class="comment">// 1: GOTO 5 ON _</span></div><div class="line">  0x415A0005, <span class="comment">// 2: GOTO 5 ON A-Z</span></div><div class="line">  0x30390005, <span class="comment">// 3: GOTO 5 ON 0-9</span></div><div class="line">  0x00FFFFFF, <span class="comment">// 4: HALT</span></div><div class="line">  0xFF000001, <span class="comment">// 5: TAKE 1</span></div><div class="line">  0x617A0005, <span class="comment">// 6: GOTO 5 ON a-z</span></div><div class="line">  0x5F5F0005, <span class="comment">// 7: GOTO 5 ON _</span></div><div class="line">  0x415A0005, <span class="comment">// 8: GOTO 5 ON A-Z</span></div><div class="line">  0x30390005, <span class="comment">// 9: GOTO 5 ON 0-9</span></div><div class="line">  0x00FFFFFF, <span class="comment">// 10: HALT</span></div><div class="line">};</div></div><!-- fragment --><p>Option <code>o</code> may be used with <code>f=machine.cpp</code> to emit optimized native C++ code for the finite state machine:</p>
<div class="fragment"><div class="line"><span class="keywordtype">void</span> reflex_code_FSM(<a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a>&amp; m)</div><div class="line">{</div><div class="line">  <span class="keywordtype">int</span> c0 = 0, c1 = 0;</div><div class="line">  m.<a class="code" href="classreflex_1_1_matcher.html#a81899695d5bb82e06e49608ed9ebd629">FSM_INIT</a>(c1);</div><div class="line"></div><div class="line">S0:</div><div class="line">  c1 = m.<a class="code" href="classreflex_1_1_matcher.html#ae2d2cbadbfa26b8b49509733971d1cc6">FSM_CHAR</a>();</div><div class="line">  <span class="keywordflow">if</span> (97 &lt;= c1 &amp;&amp; c1 &lt;= 122) <span class="keywordflow">goto</span> S5;</div><div class="line">  <span class="keywordflow">if</span> (c1 == 95) <span class="keywordflow">goto</span> S5;</div><div class="line">  <span class="keywordflow">if</span> (65 &lt;= c1 &amp;&amp; c1 &lt;= 90) <span class="keywordflow">goto</span> S5;</div><div class="line">  <span class="keywordflow">if</span> (48 &lt;= c1 &amp;&amp; c1 &lt;= 57) <span class="keywordflow">goto</span> S5;</div><div class="line">  <span class="keywordflow">return</span> m.<a class="code" href="classreflex_1_1_matcher.html#aa1d192bb300b3e62395c3662878cb054">FSM_HALT</a>(c1);</div><div class="line"></div><div class="line">S5:</div><div class="line">  m.<a class="code" href="classreflex_1_1_matcher.html#ad285214a17e5ab1b3ae763e8c7a819ef">FSM_TAKE</a>(1);</div><div class="line">  c1 = m.<a class="code" href="classreflex_1_1_matcher.html#ae2d2cbadbfa26b8b49509733971d1cc6">FSM_CHAR</a>();</div><div class="line">  <span class="keywordflow">if</span> (97 &lt;= c1 &amp;&amp; c1 &lt;= 122) <span class="keywordflow">goto</span> S5;</div><div class="line">  <span class="keywordflow">if</span> (c1 == 95) <span class="keywordflow">goto</span> S5;</div><div class="line">  <span class="keywordflow">if</span> (65 &lt;= c1 &amp;&amp; c1 &lt;= 90) <span class="keywordflow">goto</span> S5;</div><div class="line">  <span class="keywordflow">if</span> (48 &lt;= c1 &amp;&amp; c1 &lt;= 57) <span class="keywordflow">goto</span> S5;</div><div class="line">  <span class="keywordflow">return</span> m.<a class="code" href="classreflex_1_1_matcher.html#aa1d192bb300b3e62395c3662878cb054">FSM_HALT</a>(c1);</div><div class="line">}</div></div><!-- fragment --><p>The compact FSM opcode tables or the optimized larger FSM code may be used directly in your code. This omits the FSM construction overhead at runtime. Simply include this generated file in your source code and pass it on to the <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> constructor:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span>   <span class="comment">// reflex::Matcher, reflex::Pattern, reflex::Input</span></div><div class="line"><span class="preprocessor">#include &quot;machine.cpp&quot;</span> <span class="comment">// reflex_code_FSM[]</span></div><div class="line"></div><div class="line"><span class="comment">// use the pattern FSM (opcode table or C++ code) for fast search</span></div><div class="line"><span class="keyword">static</span> <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> pattern(reflex_code_FSM);</div><div class="line"></div><div class="line"><span class="comment">// use the RE/flex POSIX matcher to search for words in a string sentence</span></div><div class="line"><a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a> matcher(pattern, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>The RE/flex <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> construction options are given as a string:</p>
<table class="doxtable">
<tr>
<th>Option </th><th>Effect  </th></tr>
<tr>
<td><code>b</code> </td><td>bracket lists are parsed without converting escapes </td></tr>
<tr>
<td><code>e=c;</code> </td><td>redefine the escape character </td></tr>
<tr>
<td><code>f=file.cpp;</code> </td><td>save finite state machine code to <code>file.cpp</code> </td></tr>
<tr>
<td><code>f=file.gv;</code> </td><td>save deterministic finite state machine to <code>file.gv</code> </td></tr>
<tr>
<td><code>i</code> </td><td>case-insensitive matching, same as <code>(?i)X</code> </td></tr>
<tr>
<td><code>m</code> </td><td>multiline mode, same as <code>(?m)X</code> </td></tr>
<tr>
<td><code>n=name;</code> </td><td>use <code>reflex_code_name</code> for the machine (instead of <code>FSM</code>) </td></tr>
<tr>
<td><code>o</code> </td><td>only with option <code>f</code>: generate optimized FSM native C++ code </td></tr>
<tr>
<td><code>q</code> </td><td>Flex/Lex-style quotations "..." equal <code>\Q...\E</code>, same as <code>(?q)X</code> </td></tr>
<tr>
<td><code>r</code> </td><td>throw regex syntax error exceptions, otherwise ignore errors </td></tr>
<tr>
<td><code>s</code> </td><td>dot matches all (aka. single line mode), same as <code>(?s)X</code> </td></tr>
<tr>
<td><code>x</code> </td><td>free space mode with inline comments, same as <code>(?x)X</code> </td></tr>
<tr>
<td><code>w</code> </td><td>display regex syntax errors before raising them as exceptions </td></tr>
</table>
<p>For example, <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a> pattern(pattern, "isr")</code> enables case-insensitive dot-all matching with syntax errors thrown as <code>reflex::Pattern::Error</code> types of exceptions. By default, the <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> constructor solely throws the <code><a class="el" href="classreflex_1_1regex__error.html#a2a490e1b97afa08a8f068077a78dc63c" title="regex exceeds length limit (reflex::Pattern class only) ">reflex::regex_error::exceeds_length</a></code> and <code><a class="el" href="classreflex_1_1regex__error.html#a0c63bf6cff5be3ddb509de3ea84d6e41" title="regex exceeds complexity limits (reflex::Pattern class only) ">reflex::regex_error::exceeds_limits</a></code> exceptions and silently ignores syntax errors, see <a class="el" href="index.html#regex-pattern">The reflex::Pattern class </a>.</p>
<p>In summary:</p>
<ul>
<li>RE/flex defines an extensible abstract class interface that offers a standard API to use regex matcher engines. The API is used by the generated scanners. The API supports UTF-8/16/32-encoded FILE content, wide strings and streaming data.</li>
<li>RE/flex includes a regex matcher class and a regex pattern class to implement fast matching with deterministic finite state machines (FSMs). The FSM graph can be visualized with the Graphviz dot tool. Furthermore, this FSM can also be exported and imported as source code to expedite pattern matching.</li>
</ul>
<p><a class="el" href="index.html#regex">The RE/flex regex library </a> section has more information about the RE/flex regex library.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h1><a class="anchor" id="reflex"></a>
The RE/flex scanner generator                                         </h1>
<p>The RE/flex scanner generator <b><code>reflex</code></b> takes a lexer specification and generates a regex-based C++ lexer class that is saved to <em><code>lex.yy.cpp</code></em>, or saved to the file we specified by the <code>-o</code> command-line option. This file is then compiled and linked with option <code>-lreflex</code> (and optionally <code>-lboost_regex</code> (or <code>-lboost_regex-mt</code>) if we use Boost.Regex for matching or <code>-lpcre2-8</code> if we use PCRE2 for matching) to produce a scanner: </p><pre class="fragment">reflex lexerspec.l
c++ lex.yy.cpp -lreflex
</pre><p>We use option <code>−−header-file</code> to generate <em><code>lex.yy.h</code></em> to include in the source code of your lexer application: </p><pre class="fragment">reflex −−header-file lexerspec.l
c++ mylexer.cpp lex.yy.cpp -lreflex
</pre><p>If <code>libreflex</code> was not installed then linking with <code>-lreflex</code> fails. See <a class="el" href="index.html#link-errors">Undefined symbols and link errors </a> on how to resolve this.</p>
<p>The scanner can be a stand-alone application based on <em><code>lex.yy.cpp</code></em> alone, or be part of a larger program, such as a compiler:</p>
<div align="center">
<img src="dot_inline_dotgraph_2.png" alt="dot_inline_dotgraph_2.png" border="0" usemap="#dot_inline_dotgraph_2.map"/>
<map name="dot_inline_dotgraph_2.map" id="dot_inline_dotgraph_2.map"></map>
</div>
<p>The RE/flex-generated scanners use the RE/flex regex library API for pattern matching. The RE/flex regex library API is defined by the abstract class <code><a class="el" href="classreflex_1_1_abstract_matcher.html" title="The abstract matcher base class template defines an interface for all pattern matcher engines...">reflex::AbstractMatcher</a></code>.</p>
<p>There are three regex matching engines to choose from for the generated scanner: the Boost.Regex library, the PCRE2 linrary, or the built-in RE/flex POSIX matcher engine. In any case, the <code>libreflex</code> library should be linked. The <code>libboost_regex</code> library or the <code>libpcre2-8</code> library should only be linked when the Boost.Regex or PCRE2 engines are used for matching, respectively.</p>
<p>The input class <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> of the <code>libreflex</code> library manages input from strings, wide strings, streams, and data from <code>FILE</code> descriptors. File data may be encoded in ASCII, binary or in UTF-8/16/32. UTF-16/32 is automatically decoded and converted to UTF-8 for UTF-8-based regex matching:</p>
<div align="center">
<img src="dot_inline_dotgraph_3.png" alt="dot_inline_dotgraph_3.png" border="0" usemap="#dot_inline_dotgraph_3.map"/>
<map name="dot_inline_dotgraph_3.map" id="dot_inline_dotgraph_3.map"></map>
</div>
<p>The generated scanner executes actions (typically to produce tokens for a parser). The actions are triggered by matching patterns to the input.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-command"></a>
The reflex command line tool                                  </h2>
<p>The <b><code>reflex</code></b> command takes a lexer specification from standard input or from the specified file (usually with extension <em><code>.l</code></em>, <em><code>.ll</code></em>, <em><code>.l++</code></em>, <em><code>.lxx</code></em>, or <em><code>.lpp</code></em>) and generates a C++ scanner class that is saved to the <em><code>lex.yy.cpp</code></em> source code file.</p>
<p>The <em><code>lex.yy.cpp</code></em> source code output is structured in sections that are clean, readable, and reusable.</p>
<p>Use <b><code>reflex</code></b> option <code>−−header-file</code> to generate <em><code>lex.yy.h</code></em> to include in the source code of your application: </p><pre class="fragment">reflex −−header-file lexerspec.l
</pre><p>The <b><code>reflex</code></b> command accepts <code>−−flex</code> and <code>−−bison</code> options for compatibility with Flex and Bison/Yacc, respectively. These options allow <b><code>reflex</code></b> to be used as a replacement of the classic Flex and Lex tools: </p><pre class="fragment">reflex −−flex −−bison lexerspec.l
</pre><p>The first option <code>−−flex</code> specifies that <em><code>lexerspec.l</code></em> is a classic Flex/Lex specification with <code>yytext</code> or <code>YYText()</code> and the usual "yy" variables and functions.</p>
<p>The second option <code>−−bison</code> generates a scanner class and the usual global "yy" variables and functions such as <code>yytext</code>, <code>yyleng</code>, <code>yylineno</code>, and <code>yylex()</code> for compatibility with non-reentrant <a href="dinosaur.compilertools.net/#bison">Bison</a> parsers. See <a class="el" href="index.html#reflex-bison">Interfacing with Bison/Yacc </a> for more details on Bison parsers that are reentrant and/or use bison-bridge and bison-locations options. For Bison 3.0 C++ parsers, use <code>−−bison-cc</code> and optionally <code>−−bison-locations</code>.</p>
<p>Option <code>−−yy</code> enables both <code>−−flex</code> and <code>−−bison</code> and maximizes compatibility with Lex/Flex by generating the global <code>yyin</code> and <code>yyout</code> variables. Otherwise, <code>yyin</code> points to a <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object for advanced input handling, which is more powerful than the traditional <code>FILE*</code> type <code>yyin</code>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-options"></a>
Command-line options                                          </h2>
<p>To control the output of the <b><code>reflex</code></b> scanner generator use the command-line options described in the next subsections. These options can also be specified in the lexer specification with <em><code>%option</code></em> (or as <em><code>%o</code></em> for short):</p>
<div class="alt"> <div class="fragment"><div class="line">%option flex</div><div class="line">%option bison</div><div class="line">%option graphs-file=mygraph.gv</div></div><!-- fragment --> </div><p>The above is equivalent to the <code>−−flex</code>, <code>−−bison</code>, and <code>−−graphs-file=mygraph.gv</code> command-line options.</p>
<p>Multiple options can be grouped on a single line:</p>
<div class="alt"> <div class="fragment"><div class="line">%o flex bison graphs-file=mygraph.gv</div></div><!-- fragment --> </div><p>An option parameter name may contain hyphens (-), dots (.), and double colons (::). Flex always requires quotes with option parameters, but RE/flex does not require quotes except when special characters are used, for example:</p>
<div class="alt"> <div class="fragment"><div class="line">%o flex bison graphs-file=<span class="stringliteral">&quot;dev/output/mygraph.gv&quot;</span></div></div><!-- fragment --> </div><p>Quotes (") and backslashes (\) should be escaped in an option parameter:</p>
<div class="alt"> <div class="fragment"><div class="line">%o flex bison graphs-file=<span class="stringliteral">&quot;D:\\dev\\output\\mygraph.gv&quot;</span></div></div><!-- fragment --> </div><p>Shorter forms may be used by omitting <em><code>%o</code></em> altogether, requiring each option to be specified on a separate line:</p>
<div class="alt"> <div class="fragment"><div class="line">%flex</div><div class="line">%bison</div><div class="line">%graphs-file=<span class="stringliteral">&quot;dev/output/mygraph.gv&quot;</span></div></div><!-- fragment --> </div><p>Options that affect the regular expressions such as <em><code>%option unicode</code></em> and <em><code>%option dotall</code></em> should be specified before any regular expressions are defined and used in the specification.</p>
<p>The scanner code generated by <b><code>reflex</code></b> defines options as macros <code>REFLEX_OPTION_name</code> with a value of <code>true</code> when the option is enabled or with the value that is assigned to the option. This allows your program code to check and use RE/flex options. For example, the lexer class name is <code>REFLEX_OPTION_lexer</code> when the lexer class name is redefined with option <code>−−lexer=NAME</code>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-options-scanner"></a>
Scanner options</h3>
<h4><code>−+</code>, <code>−−flex</code></h4>
<p>This option generates a <code>yyFlexLexer</code> scanner class that is compatible with the Flex-generated <code>yyFlexLexer</code> scanner class (assuming Flex with option <code>−+</code> for C++). The generated <code>yyFlexLexer</code> class has the usual <code>yytext</code> and other "yy" variables and functions, as defined by the Flex specification standard. Without this option, RE/flex actions should be used that are lexer class methods such as <code>text()</code>, <code>echo()</code> and also the lexer's matcher methods, such as <code>matcher().more()</code>, see <a class="el" href="index.html#reflex-spec-rules">The rules section</a> for more details.</p>
<h4><code>-a</code>, <code>−−dotall</code></h4>
<p>This option makes dot (<code>.</code>) in patterns match newline. Normally dot matches a single character except a newline (<code>\n</code> ASCII 0x0A).</p>
<h4><code>-B</code>, <code>−−batch</code></h4>
<p>This option generates a batch input scanner that reads the entire input all at once when possible. This scanner is fast, but consumes more memory depending on the input data size.</p>
<h4><code>-f</code>, <code>−−full</code></h4>
<p>(RE/flex matcher only). This option adds the FSM to the generated code as a static opcode table, thus generating the scanner in full. FSM construction overhead is eliminated when the scanner is initialized, resulting in a scanner that starts scanning the input immediately. This option has no effect when option <code>−−fast</code> is specified.</p>
<h4><code>-F</code>, <code>−−fast</code></h4>
<p>(RE/flex matcher only). This option adds the FSM to the generated code as optimized native C++ code. FSM construction overhead is eliminated when the scanner is initialized, resulting in a scanner that starts scanning the input immediately. The generated code takes more space compared to the <code>−−full</code> option.</p>
<h4><code>-S</code>, <code>−−find</code></h4>
<p>This option generates a search engine to find pattern matches to invoke actions corresponding to matching patterns. Unmatched input is ignored. By contrast, option <code>-s</code> (or <code>−−nodefault</code>) produces an error when non-matching input is found.</p>
<h4><code>-i</code>, <code>−−case-insensitive</code></h4>
<p>This option ignores case in patterns. Patterns match lower and upper case letters in the ASCII range only.</p>
<h4><code>-I</code>, <code>−−interactive</code>, <code>−−always-interactive</code></h4>
<p>This option generates an interactive scanner and permits console input by sacrificing speed. By contrast, the default buffered input strategy is more efficient.</p>
<h4><code>−−indent</code> and <code>−−noindent</code></h4>
<p>This option enables or disables support for indentation matching with anchors <code>\i</code>, <code>\j</code>, and <code>\k</code>. Indentation matching is enabled by default. Matching speed may be improved by disabling indentation matching, but should only be disabled when none of the indentation anchors is used in any of the patterns.</p>
<h4><code>-m reflex</code>, <code>−−matcher=reflex</code></h4>
<p>This option generates a scanner that uses the RE/flex <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> class with a POSIX matcher engine. This is the default matcher for scanning. This option is best for Flex compatibility. This matcher supports lazy quantifiers, <a class="el" href="index.html#reflex-pattern-unicode">Unicode mode</a>, <a class="el" href="index.html#reflex-pattern-anchor">Anchors and boundaries</a>, <a class="el" href="index.html#reflex-pattern-dents">Indent/nodent/dedent</a> matching, and supports FSM output for visualization with Graphviz.</p>
<h4><code>-m boost</code>, <code>−−matcher=boost</code></h4>
<p>This option generates a scanner that uses the <code><a class="el" href="classreflex_1_1_boost_posix_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost POSIX regex matching. ">reflex::BoostPosixMatcher</a></code> class with a Boost.Regex POSIX matcher engine for scanning. The matcher supports Unicode, word boundary anchors, and more, but not lazy quantifiers. Graphviz output is not supported.</p>
<h4><code>-m boost-perl</code>, <code>−−matcher=boost-perl</code></h4>
<p>This option generates a scanner that uses the <code><a class="el" href="classreflex_1_1_boost_perl_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost Perl regex matching. ">reflex::BoostPerlMatcher</a></code> class with a Boost.Regex normal (Perl) matcher engine for scanning. The matching behavior differs from the POSIX <em>leftmost longest rule</em> and results in the first matching rule to be applied instead of the rule that produces the longest match. Graphviz output is not supported.</p>
<h4><code>-m pcre2-perl</code>, <code>−−matcher=pcre2-perl</code></h4>
<p>This option generates a scanner that uses the <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html" title="PCRE2 JIT-optimized matcher engine class implements reflex::PatternMatcher pattern matching interface...">reflex::PCRE2Matcher</a></code> class with a PCRE2 (Perl) matcher engine for scanning. The matching behavior differs from the POSIX <em>leftmost longest rule</em> and results in the first matching rule to be applied instead of the rule that produces the longest match. Graphviz output is not supported.</p>
<h4><code>−−pattern=NAME</code></h4>
<p>This option defines a custom pattern class <code>NAME</code> for the custom matcher specified with option <code>-m</code>.</p>
<h4><code>−−include=FILE</code></h4>
<p>This option defines a custom include <em><code>FILE.h</code></em> to include for the custom matcher specified with option <code>-m</code>.</p>
<h4><code>-T N</code>, <code>−−tabs=N</code></h4>
<p>This option sets the default tab size to <code>N</code>, where <code>N</code> is 1, 2, 4, or 8. The tab size is used internally to determine the column position for <a class="el" href="index.html#reflex-pattern-dents">Indent/nodent/dedent</a> matching and to determine the column position returned by <code>columno()</code>, <code>columno_end()</code>, and the number of columns returned by <code>columns()</code>. It has no effect otherwise. This option assigns the <code>T=N</code> value of the <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> constructor options at runtime. The value may be set at runtime with <code>matcher().tabs(N)</code> with <code>N</code> 1, 2, 4, or 8.</p>
<h4><code>-u</code>, <code>−−unicode</code></h4>
<p>This option makes <code>.</code>, <code>\s</code>, <code>\w</code>, <code>\l</code>, <code>\u</code>, <code>\S</code>, <code>\W</code>, <code>\L</code>, <code>\U</code> match Unicode. Also groups UTF-8 sequences in the regex, such that each UTF-8 encoded character in a regex is properly matched as one wide character.</p>
<h4><code>-x</code>, <code>−−freespace</code></h4>
<p>This option switches the <b><code>reflex</code></b> scanner to <em>free space mode</em>. Regular expressions in free space mode may contain spacing to improve readability. Spacing within regular expressions is ignored, so use <code>" "</code> or <code>[ ]</code> to match a space and <code>\h</code> to match a space or a tab character. Actions in free space mode MUST be placed in <em><code>{ }</code></em> blocks and user code must be placed in <em><code>%{ %}</code></em> blocks. Patterns ending in an escape <code>\</code> continue on the next line.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-options-files"></a>
Output files options</h3>
<h4><code>-o FILE</code>, <code>−−outfile=FILE</code></h4>
<p>This option saves the scanner to <code>FILE</code> instead of <em><code>lex.yy.cpp</code></em>.</p>
<h4><code>-t</code>, <code>−−stdout</code></h4>
<p>This option writes the scanner to stdout instead of to <em><code>lex.yy.cpp</code></em>.</p>
<h4><code>−−graphs-file[=FILE]</code></h4>
<p>(RE/flex matcher only). This option generates a Graphviz file <em><code>FILE.gv</code></em>, where <code>FILE</code> is optional. When <code>FILE</code> is omitted the <b><code>reflex</code></b> command generates the file <em><code>reflex.S.gv</code></em> for each start condition state <em><code>S</code></em> defined in the lexer specification. This includes <em><code>reflex.INITIAL.gv</code></em> for the <em><code>INITIAL</code></em> start condition state. This option can be used to visualize the RE/flex matcher's finite state machine with the <a href="http://www.graphviz.org">Graphviz dot</a> tool. For example:</p>
<div align="center">
<img src="dot_inline_dotgraph_4.png" alt="dot_inline_dotgraph_4.png" border="0" usemap="#dot_inline_dotgraph_4.map"/>
<map name="dot_inline_dotgraph_4.map" id="dot_inline_dotgraph_4.map"></map>
</div>
<p>In case you are curious: the specification for this FSM digraph has two patterns: [1] a pattern to match dollar amounts with the regex <code>\$\d+(\.\d{2})?</code> and [2] the regex <code>.|\n</code> to skip a character and advance to the next match.</p>
<h4><code>−−header-file[=FILE]</code></h4>
<p>This option generates a C++ header file <em><code>FILE.h</code></em> that declares the lexer class, in addition to the generated lexer class code, where <code>FILE</code> is optional. When <code>FILE</code> is omitted the <b><code>reflex</code></b> command generates <em><code>lex.yy.h</code></em>.</p>
<h4><code>−−regexp-file[=FILE]</code></h4>
<p>This option generates a text file <em><code>FILE.txt</code></em> that contains the scanner's regular expression patterns, where <code>FILE</code> is optional. When <code>FILE</code> is omitted the <b><code>reflex</code></b> command generates <em><code>reflex.S.txt</code></em> for each start condition state <em><code>S</code></em>. The regular expression patterns are converted from the lexer specification and translated into valid C++ strings that can be used with a regex library for pattern matching.</p>
<h4><code>−−tables-file[=FILE]</code></h4>
<p>(RE/flex matcher only). This option generates a C++ file <em><code>FILE.cpp</code></em> with the finite state machine in source code form, where <code>FILE</code> is optional. When <code>FILE</code> is omitted the <b><code>reflex</code></b> command generates <em><code>reflex.S.cpp</code></em> for each start condition state <em><code>S</code></em>. This includes the file <em><code>reflex.INITIAL.cpp</code></em> for the <em><code>INITIAL</code></em> start condition state. When this option is specified in combination with <code>−−full</code> or <code>−−fast</code>, the <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> is instantiated with the code table defined in this file. Therefore, when we combine this option with <code>−−full</code> or <code>−−fast</code> then you should compile the generated table file with the scanner. Options <code>−−full</code> and <code>−−fast</code> eliminate the FSM construction overhead when the scanner is initialized.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-options-code"></a>
Output code options</h3>
<h4><code>−−namespace=NAME</code></h4>
<p>This option places the generated scanner class in the C++ namespace <code>NAME</code> scope, that is <code>NAME::Lexer</code> (and <code>NAME::yyFlexLexer</code> when option <code>−−flex</code> is used). <code>NAME</code> can be a list of nested namespaces of the form <code>NAME1::NAME2::NAME3</code> ... or by using a dot as in <code>NAME1.NAME2.NAME3</code> ...</p>
<h4><code>−−lexer=NAME</code></h4>
<p>This option defines the <code>NAME</code> of the generated scanner class and replaces the default name <code>Lexer</code> (and replaces <code>yyFlexLexer</code> when option <code>−−flex</code> is specified). The scanner class members may be declared within a <em><code>%class{ }</code></em> block. The scanner class constructor code may be defined within a <em><code>%init{ }</code></em> block. Additional constructor arguments may be declared with <em><code>%option ctorarg="argument, argument, ...</code></em>.</p>
<h4><code>−−lex=NAME</code></h4>
<p>This option defines the <code>NAME</code> of the generated scanner function to replace the function name <code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> (and <code>yylex()</code> when option <code>−−flex</code> is specified).</p>
<h4><code>−−class=NAME</code></h4>
<p>This option defines the <code>NAME</code> of the user-defined scanner class that should be derived from the generated base <code>Lexer</code> class. Use this option when defining your own scanner class named <code>NAME</code>. You can declare your custom lexer class in the first section of the lexer specification. Because the custom lexer class is user-defined, <b><code>reflex</code></b> generates the implementation of the <code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> scanner function for this specified class.</p>
<h4><code>−−yyclass=NAME</code></h4>
<p>This option combines options <code>−−flex</code> and <code>−−class=NAME</code>.</p>
<h4><code>−−main</code></h4>
<p>This option generates a <code>main</code> function to create a stand-alone scanner that scans data from standard input (using <code>stdin</code>).</p>
<h4><code>-L</code>, <code>−−noline</code></h4>
<p>This option suppresses the <code>#line</code> directives in the generated scanner code.</p>
<h4><code>-P NAME</code>, <code>−−prefix=NAME</code></h4>
<p>This option specifies <code>NAME</code> as a prefix for the generated <code>yyFlexLexer</code> class to replace the default <code>yy</code> prefix. Also renames the prefix of <code>yylex()</code>. Generates <em><code>lex.NAME.cpp</code></em> file instead of <em><code>lex.yy.cpp</code></em>, and generates <em><code>lex.NAME.h</code></em> with option <code>−−header-file</code>.</p>
<h4><code>−−nostdinit</code></h4>
<p>This option initializes input to <code>std::cin</code> instead of using <code>stdin</code>. Automatic UTF decoding is not supported. Use <code>stdin</code> for automatic UTF BOM detection and UTF decoding of standard input streams, not <code>std::cin</code>.</p>
<h4><code>−−bison</code></h4>
<p>This option generates a scanner that works with Bison parsers, by defining global (i.e. non-thread-safe and non-reentrant) "yy" variables and functions, such as <code>yytext</code>, <code>yyleng</code>, <code>yylineno</code>, and <code>yylex()</code>. See <a class="el" href="index.html#reflex-bison">Interfacing with Bison/Yacc </a> for more details. Use option <code>−−noyywrap</code> to remove the dependency on the global <code>yywrap()</code> function. Use option <code>−−bison-locations</code> to support the Bison <em><code>%locations</code></em> feature. See also the <code>−−yy</code> option.</p>
<h4><code>−−bison-bridge</code></h4>
<p>This option generates a scanner that works with Bison pure (reentrant thead-safe) parsers using a Bison bridge for one ore more scanner objects. Combine this option with <code>−−bison-locations</code> to support the Bison <em><code>%locations</code></em> feature. See <a class="el" href="index.html#reflex-bison-bridge">Bison-bridge</a> for more details.</p>
<h4><code>−−bison-cc</code></h4>
<p>This option generates a scanner that works with Bison 3.0 <em><code>%skeleton "lalr1.cc"</code></em> C++ parsers that are thread-safe. Combine this option with <code>−−bison-locations</code> to support the Bison <em><code>%locations</code></em> grammar. See <a class="el" href="index.html#reflex-bison-cc">Bison-cc</a> for more details.</p>
<h4><code>−−bison-cc-namespace=NAME</code></h4>
<p>This option specifies one or more <code>NAME</code> namespace(s) for the Bison 3.0 <em><code>%skeleton "lalr1.cc"</code></em> C++ parser, which is <code>yy</code> by default.</p>
<h4><code>−−bison-cc-parser=NAME</code></h4>
<p>This option specifies the class <code>NAME</code> of the Bison 3.0 <em><code>%skeleton "lalr1.cc"</code></em> C++ parser, which is <code>parser</code> by default.</p>
<h4><code>−−bison-complete</code></h4>
<p>This option generates a ascnner that works with Bison 3.2 C++ complete symbols, specified by <em><code>%define api.value.type variant</code></em> and <em><code>%define api.token.constructor</code></em> in a Bison grammar file. This option also sets option <code>−−bison-cc</code> and sets <code>−−token-type</code> to the parser's <code>symbol_type</code>, and sets <code>−−token-eof</code> to <code>0</code>, assuming these options are not specified already. Combine this option with <code>−−bison-locations</code> to support the Bison <em><code>%locations</code></em> feature. See <a class="el" href="index.html#reflex-bison-complete">Bison-complete</a> for more details.</p>
<h4><code>−−bison-locations</code></h4>
<p>This option generates a scanner that works with Bison with locations enabled. See <a class="el" href="index.html#reflex-bison-locations">Bison-locations</a> for more details.</p>
<h4><code>-R</code>, <code>−−reentrant</code></h4>
<p>This option generates additional Flex-compatible <code>yylex()</code> reentrant scanner functions. RE/flex scanners are always reentrant, assuming that <em><code>%class</code></em> variables are used instead of the user declaring global variables. This is a Flex-compatibility option and should only be used with options <code>−−flex</code> and <code>−−bison</code>. With this option enabled, most Flex functions take a <code>yyscan_t</code> scanner as an extra last argument. See <a class="el" href="index.html#reflex-reentrant">Reentrant scanners</a> and also <a class="el" href="index.html#reflex-bison">Interfacing with Bison/Yacc </a>.</p>
<h4><code>-y</code>, <code>−−yy</code></h4>
<p>This option enables <code>−−flex</code> and <code>−−bison</code> to generate a scanner that accepts Flex lexer specifications and works with Bison parsers, by defining global (i.e. non-thread-safe and non-reentrant) "yy" variables and functions, such as <code>yyin</code>, <code>yyout</code>, <code>yytext</code>, <code>yyleng</code>, <code>yylineno</code>, and <code>yylex()</code>.</p>
<h4><code>−−yywrap</code> and <code>−−noyywrap</code></h4>
<p>Option <code>−−yywrap</code> generates a scanner that calls the global <code>int yywrap()</code> function when EOF is reached. This option is only applicable when <code>−−flex</code> is used for compatibility and when <code>−−flex</code> and <code>−−bison</code> are used together, or when <code>−−yy</code> is specified. Wrapping is enabled by default. Use <code>−−noyywrap</code> to disable the dependence on this global function. This option has no effect for C++ lexer classes, which have a virtual <code>int wrap()</code> (or <code>yywrap()</code> with option <code>−−flex</code>) method that may be overridden.</p>
<h4><code>−−exception=VALUE</code></h4>
<p>This option defines the exception to be thrown by the generated scanner's default rule when no rule matches the input. This option generates a default rule with action <code>throw VALUE</code> and replaces the standard default rule that echoes all unmatched input text when no rule matches. This option has no effect when option <code>-S</code> (or <code>−−find</code>) is specified. See also option <code>-s</code> (or <code>−−nodefault</code>). Care should be taken to advance the input explicitly in the exception handler, for example by calling <code>lexer.matcher().winput()</code> when <code>lexer.size()</code> is zero.</p>
<h4><code>−−token-type=NAME</code></h4>
<p>This option specifies the type of the token values returned by <code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> and <code>yylex()</code>. The type of the values returned by <code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> and <code>yylex()</code> is <code>int</code> by default. This option may be used to specify an alternate token type. Option <code>−−bison-complete</code> automatically defines the appropriate token type <code>symbol_type</code> depending the the parameters specified with options <code>−−bison-cc-namespace</code> and <code>−−bison-cc-parser</code>.</p>
<h4><code>−−token-eof=VALUE</code></h4>
<p>This option specifies the value returned by <code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> and <code>yylex()</code> when the end of the input is reached and when no <code>&lt;&lt;EOF&gt;&gt;</code> rule is present. By default, a default-constructed token type value is returned when the end of input is reached. For <code>int</code> this is <code>int()</code>, which is zero. By setting <code>−−token-type=EOF</code> the value <code>EOF</code> is returned, for example.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-options-debug"></a>
Debugging options</h3>
<h4><code>-d</code>, <code>−−debug</code></h4>
<p>This enables debug mode in the generated scanner. Running the scanner produces debug messages on <code>std::cerr</code> standard error and the <code>debug()</code> function returns nonzero. To temporarily turn off debug messages, use <code>set_debug(0)</code> in your action code. To turn debug messages back on, use <code>set_debug(1)</code>. The <code>set_debug()</code> and <code>debug()</code> methods are virtual methods of the lexer class, so you can override their behavior in a derived lexer class. This option also enables assertions that check for internal errors. See <a class="el" href="index.html#reflex-debug">Debugging and profiling </a> for details.</p>
<h4><code>-p</code>, <code>−−perf-report</code></h4>
<p>This enables the collection and reporting of statistics by the generated scanner. The scanner reports the performance statistics on <code>std::cerr</code> when EOF is reached. If your scanner does not reach EOF, then invoke the lexer's <code>perf_report()</code> method explicitly in your code. Invoking this method also resets the statistics and timers, meaning that this method will report the statistics collected since it was last called. See <a class="el" href="index.html#reflex-debug">Debugging and profiling </a> for details.</p>
<h4><code>-s</code>, <code>−−nodefault</code></h4>
<p>This suppresses the default rule that echoes all unmatched input text when no rule matches. With the <code>−−flex</code> option, the scanner reports "scanner jammed" when no rule matches by calling <code>yyFlexLexer::LexerError("scanner jammed")</code>. Without the <code>−−flex</code> and <code>−−debug</code> options, a <code>std::runtime</code> exception is raised by invoking <code>AbstractLexer::lexer_error("scanner jammed")</code>. To throw a custom exception instead, use option <code>−−exception</code> or override the virtual method <code>lexer_error</code> in a derived lexer class. The virtual methods <code>LexerError</code> and <code>lexer_error</code> may be redefined by a user-specified derived lexer class, see <a class="el" href="index.html#reflex-inherit">Inheriting Lexer/yyFlexLexer </a>. Without the <code>−−flex</code> option, but with the <code>−−debug</code> option, the default rule is suppressed without invoking <code>lexer_error</code> to raise an exception. See also options <code>−−exception=VALUE</code> and <code>-S</code> (or <code>−−find</code>).</p>
<h4><code>-v</code>, <code>−−verbose</code></h4>
<p>This displays a summary of scanner statistics.</p>
<h4><code>-w</code>, <code>−−nowarn</code></h4>
<p>This disables warnings.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-options-miscellaneous"></a>
Miscellaneous options</h3>
<h4><code>-h</code>, <code>−−help</code></h4>
<p>This displays helpful information about <b><code>reflex</code></b>.</p>
<h4><code>-V</code>, <code>−−version</code></h4>
<p>This displays the current <b><code>reflex</code></b> release version.</p>
<h4><code>−−yylineno</code>, <code>−−yymore</code></h4>
<p>These options are enabled by default and have no effect.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-spec"></a>
Lexer specifications                                             </h2>
<p>A lexer specification consists of three sections that are divided by <em><code>%%</code></em> delimiters that are placed on a single line:</p>
<div class="alt"> <div class="fragment"><div class="line">Definitions</div><div class="line">%%</div><div class="line">Rules</div><div class="line">%%</div><div class="line">User code</div></div><!-- fragment --> </div><p><a class="el" href="index.html#reflex-spec-defs">The definitions section</a> is used to define named regex patterns, to set options for the scanner, and for including C++ declarations.</p>
<p><a class="el" href="index.html#reflex-spec-rules">The rules section</a> is the main workhorse of the scanner and consists of patterns and actions, where patterns may use named regex patterns that are defined in <a class="el" href="index.html#reflex-spec-defs">The definitions section</a>. The actions are executed when patterns match. For example, the following lexer specification replaces all occurrences of <code>cow</code> by <code>chick</code> in the input to the scanner:</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line">cow      out() &lt;&lt; <span class="stringliteral">&quot;chick&quot;</span>;</div><div class="line">%%</div></div><!-- fragment --> </div><p>The default rule is to echo any input character that is read from input that does not match a rule in <a class="el" href="index.html#reflex-spec-rules">The rules section</a>, so all other text is faithfully reproduced by this simple scanner example.</p>
<p>Because the pattern <code>cow</code> also matches words partly we get <code>chicks</code> for <code>cows</code>. But we also get badly garbled output for words such as <code>coward</code> and we are skipping capitalized Cows. We can improve this with a pattern that anchors word boundaries and accepts a lower or upper case C:</p>
<div class="alt"> <div class="fragment"><div class="line">cow      \&lt;[Cc]ow\&gt;</div><div class="line">%%</div><div class="line">{cow}    out() &lt;&lt; text()[0] &lt;&lt; <span class="stringliteral">&quot;hick&quot;</span>;</div><div class="line">%%</div></div><!-- fragment --> </div><p>Note that we defined a named pattern <code>cow</code> in <a class="el" href="index.html#reflex-spec-defs">The definitions section</a> to match the start and end of a "cow" or capitalized "Cow" with the regex <code>\&lt;[Cc]ow\&gt;</code>. We use <code>{cow}</code> in our rule for matching. The matched text first character is emitted with <code>text()[0]</code> and we simply append a <code>"hick"</code> to complete our chick.</p>
<p>Note that regex grouping with parenthesis to capture text matched by a parenthesized sub-regex is generally not supported by scanner generators, so we have to use the entire matched <code>text()</code> string.</p>
<p>Flex and Lex do not support word boundary anchors <code>\&lt;</code>, <code>\&gt;</code>, <code>\b</code>, and <code>\B</code> so this example only works with RE/flex.</p>
<p>If you are wondering about the action code in our example not exactly reflecting the C code expected with Flex, then rest assured that RE/flex supports the classic Flex and Lex actions such as <code>yytext</code> instead of <code>text()</code> and <code>*yyout</code> instead of <code>out()</code>. Simply use option <code>−−flex</code> to regress to the C-style Flex names and actions. Use options <code>−−flex</code> and <code>−−bison</code> (or option <code>−−yy</code>) to regress even further to generated a global <code>yylex()</code> function and "yy" variables.</p>
<p>To create a stand-alone scanner, we add <code>main</code> to the User code section:</p>
<div class="alt"> <div class="fragment"><div class="line">cow      \&lt;[Cc]ow\&gt;</div><div class="line">%%</div><div class="line">{cow}    out() &lt;&lt; text()[0] &lt;&lt; <span class="stringliteral">&quot;hick&quot;</span>;</div><div class="line">%%</div><div class="line"><span class="keywordtype">int</span> <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a>() { <span class="keywordflow">return</span> Lexer().lex(); }</div></div><!-- fragment --> </div><p>The main function instantiates the lexer class and invokes the scanner, which will not return until the entire input is processed. In fact, you can let <b><code>reflex</code></b> generate this main function for you with option <code>−−main</code>.</p>
<p>More details on these three lexer specification sections is presented next.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-spec-defs"></a>
The definitions section</h3>
<p>The Definitions section includes name-pattern pairs to define names for patterns. Named patterns can be referenced in regex patterns by embracing them in <em><code>{</code></em> and <em><code>}</code></em>.</p>
<p>The following example defines two names for two patterns, where the second regex pattern uses the previously named pattern:</p>
<div class="alt"> <div class="fragment"><div class="line">digit     [0-9]</div><div class="line">number    {digit}+</div></div><!-- fragment --> </div><p>Patterns ending in an escape <code>\</code> continue on the next line with optional line indentation. This permits you to organize your layout of long patterns. See also <a class="el" href="index.html#reflex-pattern-freespace">Free space mode</a> to improve pattern readability.</p>
<p>Names must be defined before being referenced. Names are expanded as macros in regex patterns. For example, <code>{digit}+</code> is expanded into <code>[0-9]+</code>.</p>
<dl class="section note"><dt>Note</dt><dd>This is not entirely accurate: when a name is expanded to its right-hand side pattern <code>φ</code> then the expanded pattern <code>φ</code> is placed in a non-capturing group <code>(?:φ)</code> to preserve its structure. For example, <code>{number}</code> expands to <code>(?:{digit}+)</code> which in turn expands to <code>(?:(?:[0-9])+)</code>.</dd></dl>
<p>To inject code into the generated scanner, indent the code or place the code within a <em><code>%{ %}</code></em> block. The <em><code>%{</code></em> and the matching <em><code>%}</code></em> should each be placed at the start of a new line. To inject code at the very top of the generated scanner, place this code within a <em><code>%top{ }</code></em> block:</p>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &lt;iostream&gt;</span>    <span class="comment">// std::cout etc.</span></div><div class="line">}</div></div><!-- fragment --> </div><p><a class="el" href="index.html#reflex-spec-defs">The definitions section</a> may also contain one or more options with <em><code>%option</code></em> (or <em><code>%o</code></em> for short). For example:</p>
<div class="alt"> <div class="fragment"><div class="line">%option <a class="code" href="namespacereflex_1_1convert__flag.html#ab6a63c3a106c2dbcbad7b86797453f58">dotall</a> <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a></div><div class="line">%o matcher=boost</div></div><!-- fragment --> </div><p>Multiple options can be grouped on the same line as is shown above. See <a class="el" href="index.html#reflex-options">Command-line options </a> for a list of available options.</p>
<dl class="section note"><dt>Note</dt><dd>Options <code>freespace</code>, <code>case-insensitive</code>, <code>dotall</code>, and <code>unicode</code> affect the named patterns defined in <a class="el" href="index.html#reflex-spec-defs">The definitions section</a>. Therefore, we should place these options ahead of all named patterns. If a regex pattern specifically requires one or more of these options, use the <code>(?isux:φ)</code> modifier(s), see <a class="el" href="index.html#reflex-patterns">Patterns </a> for details.</dd></dl>
<p>Consider the following example. Say we want to count the number of occurrences of the word "cow" in some text. We declare a global counter, increment the counter when we see a "cow", and finally report the total tally when we reach the end of the input marked by the <code>&lt;&lt;EOF&gt;&gt;</code> rule:</p>
<div class="alt"> <div class="fragment"><div class="line">%option <a class="code" href="namespacereflex_1_1convert__flag.html#ab6a63c3a106c2dbcbad7b86797453f58">dotall</a> <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a></div><div class="line"></div><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &lt;iostream&gt;</span>    <span class="comment">// std::cout etc.</span></div><div class="line">}</div><div class="line"></div><div class="line">%{</div><div class="line">  <span class="keyword">static</span> <span class="keywordtype">int</span> herd = 0;   <span class="comment">// a global static variable to count cows</span></div><div class="line">%}</div><div class="line"></div><div class="line">cow        \&lt;[Cc]ow\&gt;</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">{cow}      herd++;       <span class="comment">// found a cow, bump count by one</span></div><div class="line">.          <span class="comment">// do nothing</span></div><div class="line">&lt;&lt;EOF&gt;&gt;    out() &lt;&lt; herd &lt;&lt; <span class="stringliteral">&quot; cows!&quot;</span> &lt;&lt; std::endl; <span class="keywordflow">return</span> 0;</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>The above works fine, but we are using a global counter which is not a best practice and is not thread-safe: multiple Lexer class instances may compete to bump the counter. Another problem is that the Lexer can only be used once, there is no proper initialization to restart the Lexer on new input.</p>
<p>RE/flex allows you to inject code in the generated Lexer class, meaning that class members and constructor code can be added to manage the Lexer class state. All Lexer class members are visible in actions, even when private. New Lexers can be instantiated given some input to scan. Lexers can run in parallel in threads without requiring synchronization when their state is part of the instance and not managed by global variables.</p>
<p>To inject Lexer class member declarations such as variables and methods, place the declarations within <em><code>%class{ }</code></em> block. The <em><code>%class{</code></em> and the matching <em><code>}</code></em> should each be placed at the start of a new line.</p>
<p>Likewise, to inject Lexer class constructor code, for example to initialize members, place the code within <em><code>%init{ }</code></em> block. The <em><code>%init{</code></em> and the matching <em><code>}</code></em> should each be placed at the start of a new line. Option <em><code>%option ctorarg="argument, argument, ..."</code></em> may be used to declare the constructor arguments of the Lexer class constructor.</p>
<p>Additional constructors and/or a destructor may be placed in a <em><code>%class{ }</code></em> block, for class <code>Lexer</code> (or <code>yyFlexLexer</code> with option <code>−−flex</code>), unless the class is renamed with option <code>−−lexer=NAME</code> (<em><code>%option lexer=NAME</code></em>).</p>
<p>For convenience you can use the generated <code>REFLEX_OPTION_lexer</code> macro in your code that expands to the class name. To do so, use <b><code>reflex</code></b> option <code>−−header-file</code> to generate a header file to include in your code.</p>
<p>For example, we use these code injectors to make our cow counter <code>herd</code> part of the Lexer class state:</p>
<div class="alt"> <div class="fragment"><div class="line">%option <a class="code" href="namespacereflex_1_1convert__flag.html#ab6a63c3a106c2dbcbad7b86797453f58">dotall</a> <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a></div><div class="line"></div><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &lt;iostream&gt;</span>    <span class="comment">// std::cout etc.</span></div><div class="line">}</div><div class="line"></div><div class="line">%<span class="keyword">class</span>{</div><div class="line">  <span class="keywordtype">int</span> herd;  <span class="comment">// lexer class member variable (private by default)</span></div><div class="line">}</div><div class="line"></div><div class="line">%init{</div><div class="line">  herd = 0;  <span class="comment">// initialize member variable in Lexer class constructor</span></div><div class="line">}</div><div class="line"></div><div class="line">cow        \&lt;[Cc]ow\&gt;</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">{cow}      herd++;       <span class="comment">// found a cow, bump count by one</span></div><div class="line">.          <span class="comment">// do nothing</span></div><div class="line">&lt;&lt;EOF&gt;&gt;    out() &lt;&lt; herd &lt;&lt; <span class="stringliteral">&quot; cows!&quot;</span> &lt;&lt; std::endl; <span class="keywordflow">return</span> 0;</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>Note that nothing else needed to be changed, because the actions are part of the generated Lexer class and can access the Lexer class members, in this example the member variable <code>herd</code>.</p>
<p>To modularize specifications of lexers, use <em><code>%include</code></em> (or <em><code>%i</code></em> for short) to include one or more files into <a class="el" href="index.html#reflex-spec-defs">The definitions section</a> of a specification. For example:</p>
<div class="alt"> <div class="fragment"><div class="line">%include <span class="stringliteral">&quot;examples/jdefs.l&quot;</span></div></div><!-- fragment --> </div><p>This includes <em><code>examples/jdefs.l</code></em> with Java patterns into the current specification so you can match Java lexical structures, such as copying Java identifiers to the output given some Java source program as input:</p>
<div class="alt"> <div class="fragment"><div class="line">%include <span class="stringliteral">&quot;examples/jdefs.l&quot;</span></div><div class="line">%%</div><div class="line">{Identifier}    echo();</div><div class="line">.|\n            <span class="comment">// do nothing</span></div><div class="line">%%</div></div><!-- fragment --> </div><p>Multiple files may be specified with one <em><code>%include</code></em>. Quotes may be omitted from the <em><code>%include</code></em> argument if the argument has no punctuation characters except <code>.</code> and <code>-</code>, for example <em><code>%include jdefs.l</code></em>.</p>
<p>To declare start condition state names use <em><code>%state</code></em> (or <em><code>%s</code></em> for short) to declare inclusive states and use <em><code>%xstate</code></em> (or <em><code>%x</code></em> for short) to declare exclusive states:</p>
<div class="alt"> <div class="fragment"><div class="line">%s INCLUSIVE</div><div class="line">%x EXCLUSIVE</div></div><!-- fragment --> </div><p>See <a class="el" href="index.html#reflex-states">Start condition states </a> for more information about states.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-spec-rules"></a>
The rules section</h3>
<p>Each rule in the rules section consists of a pattern-action pair separated by spacing after the pattern (unless free space mode is enabled). For example, the following defines an action for a pattern:</p>
<div class="alt"> <div class="fragment"><div class="line">[0-9]+      out() &lt;&lt; <span class="stringliteral">&quot;number &quot;</span> &lt;&lt; text() &lt;&lt; std::endl;</div></div><!-- fragment --> </div><p>To add action code that spans multiple lines, indent the code or place the code in a <em><code>{ }</code></em> block. When local variables are declared in an action then the code should always be placed in a block.</p>
<p>In free space mode you MUST place actions in <em><code>{ }</code></em> blocks and user code in <em><code>%{ %}</code></em> blocks instead of indented, see <a class="el" href="index.html#reflex-pattern-freespace">Free space mode</a>.</p>
<p>Actions in the rules section can use predefined RE/flex variables and functions. With <b><code>reflex</code></b> option <code>−−flex</code>, the variables and functions are the classic Flex actions shown in the second column of this table:</p>
<table class="doxtable">
<tr>
<th>RE/flex action </th><th>Flex action </th><th>Result  </th></tr>
<tr>
<td><code>text()</code> </td><td><code>YYText()</code>, <code>yytext</code> </td><td>0-terminated text match </td></tr>
<tr>
<td><code>str()</code> </td><td><em>n/a</em> </td><td><code>std::string</code> text match </td></tr>
<tr>
<td><code>wstr()</code> </td><td><em>n/a</em> </td><td><code>std::wstring</code> wide text match </td></tr>
<tr>
<td><code>chr()</code> </td><td><code>yytext[0]</code> </td><td>first 8-bit char of text match </td></tr>
<tr>
<td><code>wchr()</code> </td><td><em>n/a</em> </td><td>first wide char of text match </td></tr>
<tr>
<td><code>size()</code> </td><td><code>YYLeng()</code>, <code>yyleng</code> </td><td>size of the match in bytes </td></tr>
<tr>
<td><code>wsize()</code> </td><td><em>n/a</em> </td><td>number of wide chars matched </td></tr>
<tr>
<td><code>lines()</code> </td><td><em>n/a</em> </td><td>number of lines matched (&gt;=1) </td></tr>
<tr>
<td><code>columns()</code> </td><td><em>n/a</em> </td><td>number of columns matched (&gt;=0) </td></tr>
<tr>
<td><code>lineno()</code> </td><td><code>yylineno</code> </td><td>line number of match (&gt;=1) </td></tr>
<tr>
<td><code>columno()</code> </td><td><em>n/a</em> </td><td>column number of match (&gt;=0) </td></tr>
<tr>
<td><code>lineno_end()</code> </td><td><em>n/a</em> </td><td>ending line number of match (&gt;=1) </td></tr>
<tr>
<td><code>columno_end()</code> </td><td><em>n/a</em> </td><td>ending column number of match (&gt;=0) </td></tr>
<tr>
<td><code>border()</code> </td><td><em>n/a</em> </td><td>border of the match (&gt;=0) </td></tr>
<tr>
<td><code>echo()</code> </td><td><code>ECHO</code> </td><td><code>out().write(text(), size())</code> </td></tr>
<tr>
<td><code>in(i)</code> </td><td><code><a class="el" href="flexlexer_8h.html#add6f1ba7912f4e259567ebdc120bff17" title="Flex-compatible macro: restart from the given input source. ">yyrestart(i)</a></code> </td><td>set input to <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> </td></tr>
<tr>
<td><code>in()</code>, <code>in() = i</code> </td><td><code>*yyin</code>, <code>yyin = &amp;i</code> </td><td>get/set <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> </td></tr>
<tr>
<td><code>out(o)</code> </td><td><code>yyout = &amp;o</code> </td><td>set output to <code>std::ostream o</code> </td></tr>
<tr>
<td><code>out()</code> </td><td><code>*yyout</code> </td><td>get <code>std::ostream</code> object </td></tr>
<tr>
<td><code>out().write(s, n)</code> </td><td><code>LexerOutput(s, n)</code> </td><td>output chars <code>s[0..n-1]</code> </td></tr>
<tr>
<td><code>out().put(c)</code> </td><td><code>output(c)</code> </td><td>output char <code>c</code> </td></tr>
<tr>
<td><code>start(n)</code> </td><td><code>BEGIN n</code> </td><td>set start condition to <code>n</code> </td></tr>
<tr>
<td><code>start()</code> </td><td><code>YY_START</code> </td><td>get current start condition </td></tr>
<tr>
<td><code>push_state(n)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a568a847aae0d31ca8358c9c52d22f15d" title="Flex-compatible macro: push the current start condition state on the stack and transition to the give...">yy_push_state(n)</a></code> </td><td>push current state, start <code>n</code> </td></tr>
<tr>
<td><code>pop_state()</code> </td><td><code><a class="el" href="flexlexer_8h.html#a650b5dd502b5fdf13563de668b7f104f" title="Flex-compatible macro: pop the stack start condition state and transition to that state...">yy_pop_state()</a></code> </td><td>pop state and make it current </td></tr>
<tr>
<td><code>top_state()</code> </td><td><code><a class="el" href="flexlexer_8h.html#a7d315e7bf2129b110727d0deac8a2031" title="Flex-compatible macro: returns the stack top start condition state. ">yy_top_state()</a></code> </td><td>get top state start condition </td></tr>
<tr>
<td><code>states_empty()</code> </td><td><em>n/a</em> </td><td>true if state stack is empty </td></tr>
<tr>
<td><code>matcher().accept()</code> </td><td><code>yy_act</code> </td><td>number of the matched rule </td></tr>
<tr>
<td><code>matcher().text()</code> </td><td><code>YYText()</code>, <code>yytext</code> </td><td>same as <code>text()</code> </td></tr>
<tr>
<td><code>matcher().str()</code> </td><td><em>n/a</em> </td><td>same as <code>str()</code> </td></tr>
<tr>
<td><code>matcher().wstr()</code> </td><td><em>n/a</em> </td><td>same as <code>wstr()</code> </td></tr>
<tr>
<td><code>matcher().chr()</code> </td><td><code>yytext[0]</code> </td><td>same as <code>chr()</code> </td></tr>
<tr>
<td><code>matcher().wchr()</code> </td><td><em>n/a</em> </td><td>same as <code>wchr()</code> </td></tr>
<tr>
<td><code>matcher().size()</code> </td><td><code>YYLeng()</code>, <code>yyleng</code> </td><td>same as <code>size()</code> </td></tr>
<tr>
<td><code>matcher().wsize()</code> </td><td><em>n/a</em> </td><td>same as <code>wsize()</code> </td></tr>
<tr>
<td><code>matcher().lines()</code> </td><td><em>n/a</em> </td><td>same as <code>lines()</code> </td></tr>
<tr>
<td><code>matcher().columns()</code> </td><td><em>n/a</em> </td><td>same as <code>columns()</code> </td></tr>
<tr>
<td><code>matcher().lineno()</code> </td><td><code>yylineno</code> </td><td>same as <code>lineno()</code> </td></tr>
<tr>
<td><code>matcher().columno()</code> </td><td><em>n/a</em> </td><td>same as <code>columno()</code> </td></tr>
<tr>
<td><code>matcher().lineno_end()</code> </td><td><code>yylineno</code> </td><td>same as <code>lineno_end()</code> </td></tr>
<tr>
<td><code>matcher().columno_end()</code></td><td><em>n/a</em> </td><td>same as <code>columno_end()</code> </td></tr>
<tr>
<td><code>matcher().border()</code> </td><td><em>n/a</em> </td><td>same as <code>border()</code> </td></tr>
<tr>
<td><code>matcher().begin()</code> </td><td><em>n/a</em> </td><td>non-0-terminated text match begin </td></tr>
<tr>
<td><code>matcher().end()</code> </td><td><em>n/a</em> </td><td>non-0-terminated text match end </td></tr>
<tr>
<td><code>matcher().input()</code> </td><td><code><a class="el" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a" title="Flex-compatible macro: read one 8-bit character, returns zero when EOF. ">yyinput()</a></code> </td><td>get next 8-bit char from input </td></tr>
<tr>
<td><code>matcher().winput()</code> </td><td><em>n/a</em> </td><td>get wide character from input </td></tr>
<tr>
<td><code>matcher().unput(c)</code> </td><td><code>unput(c)</code> </td><td>put back 8-bit char <code>c</code> </td></tr>
<tr>
<td><code>matcher().wunput(c)</code> </td><td><code>unput(c)</code> </td><td>put back (wide) char <code>c</code> </td></tr>
<tr>
<td><code>matcher().peek()</code> </td><td><em>n/a</em> </td><td>peek at next 8-bit char on input </td></tr>
<tr>
<td><code>matcher().skip(c)</code> </td><td><em>n/a</em> </td><td>skip input to char <code>c</code> </td></tr>
<tr>
<td><code>matcher().skip(s)</code> </td><td><em>n/a</em> </td><td>skip input to UTF-8 string <code>s</code> </td></tr>
<tr>
<td><code>matcher().more()</code> </td><td><code><a class="el" href="flexlexer_8h.html#a8f5ae76ff361bbf03790c326d86ea600" title="Flex-compatible macro: append the next matched text to the currently matched text (use only when scan...">yymore()</a></code> </td><td>append next match to this match </td></tr>
<tr>
<td><code>matcher().less(n)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a6e9a1ca8eb82e1d2dea4fa4a4dfc5c07" title="Flex-compatible macro: truncate the yytext length of the match to n characters in length and repositi...">yyless(n)</a></code> </td><td>shrink match length to <code>n</code> </td></tr>
<tr>
<td><code>matcher().first()</code> </td><td><em>n/a</em> </td><td>first pos of match in input </td></tr>
<tr>
<td><code>matcher().last()</code> </td><td><em>n/a</em> </td><td>last pos+1 of match in input </td></tr>
<tr>
<td><code>matcher().rest()</code> </td><td><em>n/a</em> </td><td>get rest of input until end </td></tr>
<tr>
<td><code>matcher().span()</code> </td><td><em>n/a</em> </td><td>enlarge match to span line </td></tr>
<tr>
<td><code>matcher().line()</code> </td><td><em>n/a</em> </td><td>get line with the match </td></tr>
<tr>
<td><code>matcher().wline()</code> </td><td><em>n/a</em> </td><td>get line with the match </td></tr>
<tr>
<td><code>matcher().at_bob()</code> </td><td><em>n/a</em> </td><td>true if at the begin of input </td></tr>
<tr>
<td><code>matcher().at_end()</code> </td><td><em>n/a</em> </td><td>true if at the end of input </td></tr>
<tr>
<td><code>matcher().at_bol()</code> </td><td><code><a class="el" href="flexlexer_8h.html#a6e440c28c90cfded2ea454bd0d19616e" title="Flex-compatible macro: at-begin-of-line check (use only when scanner has started). ">YY_AT_BOL()</a></code> </td><td>true if at begin of a newline </td></tr>
<tr>
<td><code>set_debug(n)</code> </td><td><code>set_debug(n)</code> </td><td>reflex option <code>-d</code> sets <code>n=1</code> </td></tr>
<tr>
<td><code>debug()</code> </td><td><code>debug()</code> </td><td>nonzero when debugging </td></tr>
</table>
<p>A <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> input source is denoted <code>i</code> in the table, which can be <code>FILE*</code> descriptor, <code>std::istream</code>, a string <code>std::string</code> or <code>const char*</code>, or a wide string <code>std::wstring</code> or <code>const wchar_t*</code>. Output <code>o</code> is a <code>std::ostream</code> object.</p>
<p>Note that Flex <code>switch_streams(i, o)</code> is the same as invoking the <code>in(i)</code> and <code>out(o)</code> methods. Flex <code><a class="el" href="flexlexer_8h.html#add6f1ba7912f4e259567ebdc120bff17" title="Flex-compatible macro: restart from the given input source. ">yyrestart(i)</a></code> is the same as invoking <code>in(i)</code> to set input to a file, stream, or string. Invoking <code>switch_streams(i, o)</code> and <code>in(i)</code> also resets the lexer's matcher (internally with <code>matcher.reset()</code>). This clears the line and column counters, resets the internal anchor and boundary flags for anchor and word boundary matching, and resets the matcher to consume buffered input.</p>
<p>You can also set the input with <code>in() = i</code> (or <code>yyin = &amp;i)</code> with option <code>−−flex</code>). This however does not reset the matcher. This means that when an end of input (EOF) was reached, you should clear the EOF state first with <code>matcher().set_end(false)</code> or reset the matcher state with <code>matcher().reset()</code>. Resetting the matcher state also flushes the remaining input from the buffer, which would otherwise still be consumed. Using <code>in(i)</code> (or <code><a class="el" href="flexlexer_8h.html#add6f1ba7912f4e259567ebdc120bff17" title="Flex-compatible macro: restart from the given input source. ">yyrestart(i)</a></code> with option <code>−−flex</code>) is therefore preferable.</p>
<p>The <code>matcher().input()</code>, <code>matcher().winput()</code>, and <code>matcher().peek()</code> methods return a non-negative character code and EOF (-1) when the end of input is reached. These methods preserve the current <code>text()</code> match (and <code>yytext</code> with option <code>−−flex</code>), but the pointer returned by <code>text()</code> (and <code>yytext</code>) may change after these methods are called. However, the <code>yytext</code> pointer is not preserved when using these methods with <b><code>reflex</code></b> options <code>−−flex</code> and <code>−−bison</code>.</p>
<dl class="section warning"><dt>Warning</dt><dd>The Flex-compatible <code><a class="el" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a" title="Flex-compatible macro: read one 8-bit character, returns zero when EOF. ">yyinput()</a></code> function returns 0 when the end of input is reached, which makes it impossible to distinguish <code>\0</code> (NUL) from EOF. By contrast, <code>matcher().input()</code> returns EOF (-1) when the end of the input is reached.</dd>
<dd>
Do not invoke <code>matcher()</code> before the <code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> (or <code>yylex()</code> with option <code>−−flex</code>) is invoked! A matcher is not initially assigned to a lexer when the lexer is constructed, leaving <code>matcher()</code> undefined.</dd></dl>
<p>The <code>matcher().skip(c)</code> method skips input until <code>char</code> or wide <code>wchar_t</code> character <code>c</code> is consumed and returns <code>true</code> when found. This method changes <code>text()</code> (and <code>yytext</code> with option <code>−−flex</code>). This method is more efficient than repeatedly calling <code>matcher().input()</code>. Likewise, <code>matcher().skip(s)</code> skips input until UTF-8 string <code>s</code> is consumed and returns <code>true</code> when found.</p>
<p>Use <b><code>reflex</code></b> options <code>−−flex</code> and <code>−−bison</code> (or option <code>−−yy</code>) to enable global Flex actions and variables. This makes Flex actions and variables globally accessible outside of <a class="el" href="index.html#reflex-spec-rules">The rules section</a>, with the exception of <code><a class="el" href="flexlexer_8h.html#a568a847aae0d31ca8358c9c52d22f15d" title="Flex-compatible macro: push the current start condition state on the stack and transition to the give...">yy_push_state()</a></code>, <code><a class="el" href="flexlexer_8h.html#a650b5dd502b5fdf13563de668b7f104f" title="Flex-compatible macro: pop the stack start condition state and transition to that state...">yy_pop_state()</a></code>, <code><a class="el" href="flexlexer_8h.html#a7d315e7bf2129b110727d0deac8a2031" title="Flex-compatible macro: returns the stack top start condition state. ">yy_top_state()</a></code>. Outside <a class="el" href="index.html#reflex-spec-rules">The rules section</a> you must use the global action <code><a class="el" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a" title="Flex-compatible macro: read one 8-bit character, returns zero when EOF. ">yyinput()</a></code> instead of <code>input()</code>, global action <code><a class="el" href="flexlexer_8h.html#a978e5155bc5a5e66b2cadcb7b2124c72" title="Flex-compatible macro: put one 8-bit character back onto the input stream to be read again...">yyunput()</a></code> instead of <code>unput()</code>, and global action <code><a class="el" href="flexlexer_8h.html#a01e073defdea723b59e8e4415648ac47" title="Flex-compatible macro: output one 8-bit character. ">yyoutput()</a></code> instead of <code>output()</code>. Because <code>yyin</code> and <code>yyout</code> are macros they cannot be (re)declared or accessed as global variables, but they can be used as if these were variables. To avoid compilation errors, use <b><code>reflex</code></b> option <code>−−header-file</code> to generate a header file <em><code>lex.yy.h</code></em> to include in your code to use the global use Flex actions and variables. See <a class="el" href="index.html#reflex-bison">Interfacing with Bison/Yacc </a> for more details on the <code>−−bison</code> options to use.</p>
<p>When using <b><code>reflex</code></b> options <code>−−flex</code>, <code>−−bison</code> and <code>−−reentrant</code>, most Flex functions take a <code>yyscan_t</code> scanner as an extra last argument. See <a class="el" href="index.html#reflex-reentrant">Reentrant scanners</a> for details.</p>
<p>From the first couple of entries in the table shown above you may have guessed correctly that <code>text()</code> is just a shorthand for <code>matcher().text()</code>, since <code>matcher()</code> is the matcher object associated with the generated Lexer class. The same shorthand apply to <code>str()</code>, <code>wstr()</code>, <code>size()</code>, <code>wsize()</code>, <code>lineno()</code>, <code>columno()</code>, and <code>border()</code>. Use <code>text()</code> for fast access to the matched text. The <code>str()</code> method returns a string copy of the match and is less efficient. Likewise, <code>wstr()</code> returns a wide string copy of the match, converted from UTF-8.</p>
<p>The <code>lineno()</code> method returns the line number of the match, starting at line 1. The ending line number is <code>lineno_end()</code>, which is identical to the value of <code>lineno()</code> + <code>lines()</code> - 1.</p>
<p>The <code>columno()</code> method returns the column offset of a match from the start of the line, beginning at column 0. This method takes tab spacing and wide characters into account. The inclusive ending column number is given by <code>columno_end()</code>, which is equal or larger than <code>columno()</code> if the match does not span multiple lines. Otherwise, if the match spans multiple lines, <code>columno_end()</code> is the ending column of the match on the last matching line.</p>
<p>The <code>lines()</code> and <code>columns()</code> methods return the number of lines and columns matched, where <code>columns()</code> takes tab spacing and wide characters into account. If the match spans multiple lines, <code>columns()</code> counts columns over all lines, without counting the newline characters.</p>
<p>The starting byte offset of the match on a line is <code>border()</code> and the inclusive ending byte offset of the match is <code>border() + size() - 1</code>.</p>
<dl class="section note"><dt>Note</dt><dd>A wide character is counted as one, thus <code>columno()</code>, <code>columno_end()</code>, and <code>columns()</code> do not take the character width of full-width and combining Unicode characters into account. It is recommended to use the <code>wcwidth</code> function or <a href="https://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c">wcwidth.c</a> to determine Unicode character widths.</dd></dl>
<p>The <code>matcher().more()</code> method is used to create longer matches by stringing together consecutive matches in the input after scanning the input with the <code>scan()</code> method. When this method is invoked, the next match with <code>scan()</code> has its matched text prepended to it. The <code>matcher().more()</code> operation is often used in lexers and was introduced in Lex.</p>
<p>The <code>matcher().less(n)</code> method reduces the size of the matched text to <code>n</code> bytes. This method has no effect if <code>n</code> is larger than <code>size()</code>. The value of <code>n</code> should not be <code>0</code> to prevent infinite looping on the same input as no input is consumed (or you could switch to another start condition state with <code>start(n)</code> in the action that uses <code>less(0)</code>). The <code>matcher().less(n)</code> operation was introduced in Lex and is often used in lexers to place input back into the input stream and as a means to perform sophisticated lookaheads.</p>
<p>The <code>matcher().first()</code> and <code>matcher().last()</code> methods return the position in the input stream of the match, counting in bytes from the start of the input at position 0. If the input stream is a wide character sequence, the UTF-8 positions are returned as a result of the internally-converted UTF-8 wide character input.</p>
<p>The <code>matcher().rest()</code> method returns the rest of the input character sequence as a 0-terminated <code>char*</code> string. This method buffers all remaining input to return the string.</p>
<p>The <code>matcher().span()</code> method enlarges the text matched to span the entire line and returns the matching line as a 0-terminated <code>char*</code> string without the <code>\n</code>.</p>
<p>The <code>matcher().line()</code> and <code>matcher().wline()</code> methods return the entire line as a (wide) string with the matched text as a substring. These methods can be used to obtain the context of a match, for example to display the line where a lexical error or syntax error occurred.</p>
<dl class="section warning"><dt>Warning</dt><dd>The methods <code>matcher().span()</code>, <code>matcher().line()</code>, and <code>matcher().wline()</code> invalidate the previous <code>text()</code>, <code>yytext</code>, <code>begin()</code>, and <code>end()</code> string pointers. Call these methods again to retrieve the updated pointer or call <code>str()</code> or <code>wstr()</code> to obtain a string copy of the match: <div class="fragment"><div class="line"><span class="comment">// INCORRECT, because t is invalid after line():</span></div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *t = matcher().text();</div><div class="line">std::string s = matcher().line();</div><div class="line">std::cout &lt;&lt; t &lt;&lt; <span class="stringliteral">&quot; in &quot;</span> &lt;&lt; s &lt;&lt; std::endl;</div><div class="line"><span class="comment">// OK with line():</span></div><div class="line">std::string s = matcher().line();</div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *t = matcher().text();</div><div class="line">std::cout &lt;&lt; t &lt;&lt; <span class="stringliteral">&quot; in &quot;</span> &lt;&lt; s &lt;&lt; std::endl;</div><div class="line"><span class="comment">// OK with span():</span></div><div class="line">std::string t = matcher().str();</div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *s = matcher().span();</div><div class="line">std::cout &lt;&lt; t &lt;&lt; <span class="stringliteral">&quot; in &quot;</span> &lt;&lt; s &lt;&lt; std::endl;</div></div><!-- fragment --> The start of a line is truncated when the line is too long. The length of the line's contents before the pattern match on the line is restricted to 8KB, which is the size specified by <code><a class="el" href="structreflex_1_1_abstract_matcher_1_1_const.html#a6f7de58d1118b6f74f117a6b259359be" title="buffer growth factor, buffer is initially 2*BLOCK size, at least 256 ">reflex::AbstractMatcher::Const::BLOCK</a></code>. When this length is exceeded, the line's length before the match is truncated to 8KB. This ensures that pattern matching binary files or files with very long lines cannot cause memory allocation exceptions.</dd></dl>
<p>Because <code>matcher()</code> returns the current matcher object, the following Flex-like actions are also supported:</p>
<table class="doxtable">
<tr>
<th>RE/flex action </th><th>Flex action </th><th>Result  </th></tr>
<tr>
<td><code>matcher().buffer()</code> </td><td><em>n/a</em> </td><td>buffer entire input </td></tr>
<tr>
<td><code>matcher().buffer(n)</code> </td><td><em>n/a</em> </td><td>set buffer size to <code>n</code> </td></tr>
<tr>
<td><code>matcher().interactive()</code> </td><td><code><a class="el" href="flexlexer_8h.html#a1584c58cbc37bb021cbce23ddb7594c7" title="Flex-compatible macro: interactive mode on/off (use only when scanner has started). ">yy_set_interactive(1)</a></code> </td><td>set interactive input </td></tr>
<tr>
<td><code>matcher().flush()</code> </td><td><code>YY_FLUSH_BUFFER</code> </td><td>flush input buffer </td></tr>
<tr>
<td><code>matcher().get(s, n)</code> </td><td><code>LexerInput(s, n)</code> </td><td>read <code>s[0..n-1]</code> </td></tr>
<tr>
<td><code>matcher().set_bol(b)</code> </td><td><code><a class="el" href="flexlexer_8h.html#afc284777204fb44a0af09386a3516e57" title="Flex-compatible macro: a begin-of-line forced set (use only when scanner has started). ">yy_set_bol(b)</a></code> </td><td>(re)set begin of line </td></tr>
<tr>
<td><code>matcher().set_bob(b)</code> </td><td><em>n/a</em> </td><td>(re)set begin of input </td></tr>
<tr>
<td><code>matcher().set_end(b)</code> </td><td><em>n/a</em> </td><td>(re)set end of input </td></tr>
<tr>
<td><code>matcher().reset()</code> </td><td><em>n/a</em> </td><td>reset the state as new </td></tr>
</table>
<p>You can switch to a new matcher while scanning input, and use operations to create a new matcher, push/pop a matcher on/from a stack, and delete a matcher:</p>
<table class="doxtable">
<tr>
<th>RE/flex action </th><th>Flex action </th><th>Result  </th></tr>
<tr>
<td><code>matcher(m)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a97d5ac1e34fadb1c7121a3bb9837a973" title="Flex-compatible macro: switch to another buffer. ">yy_switch_to_buffer(m)</a></code> </td><td>use matcher <code>m</code> </td></tr>
<tr>
<td><code>new_matcher(i)</code> </td><td><code><a class="el" href="flexlexer_8h.html#aa28e58c21d4fbe7132b6de8edf76e43a" title="Flex-compatible macro: create and return a new buffer. ">yy_create_buffer(i, n)</a></code> </td><td>returns new matcher for <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> </td></tr>
<tr>
<td><code>del_matcher(m)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229" title="Flex-compatible macro: delete a buffer. ">yy_delete_buffer(m)</a></code> </td><td>delete matcher <code>m</code> </td></tr>
<tr>
<td><code>push_matcher(m)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a3807e0a231906457045e429d6721f291" title="Flex-compatible macro: push the current buffer on the stack to use the given buffer. ">yypush_buffer_state(m)</a></code> </td><td>push current matcher, use <code>m</code> </td></tr>
<tr>
<td><code>pop_matcher()</code> </td><td><code><a class="el" href="flexlexer_8h.html#ad0d14a86a4c2feaa7e9e73fd26239dc1" title="Flex-compatible macro: pop buffer from the stack and delete the current buffer. ">yypop_buffer_state()</a></code> </td><td>pop matcher and delete current </td></tr>
<tr>
<td><code>ptr_matcher()</code> </td><td><code>YY_CURRENT_BUFFER</code> </td><td>pointer to current matcher </td></tr>
<tr>
<td><code>has_matcher()</code> </td><td><code>YY_CURRENT_BUFFER != 0</code> </td><td>current matcher is usable </td></tr>
</table>
<p>The matcher type <code>m</code> is a Lexer class-specific <code>Matcher</code> type, which depends on the underlying matcher used by the scanner. Therefore, <code>new_matcher(i)</code> instantiates a <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> or the matcher specified with the <code>−−matcher</code> option.</p>
<p>The <code>push_matcher()</code> and <code>pop_matcher()</code> functions can be used to temporarily switch to another input source while preserving the original input source associated with the matcher on the stack with <code>push_matcher()</code>. The <code>pop_matcher()</code> action returns <code>true</code> when successful and <code>false</code> otherwise, when the stack is empty. When <code>false</code>, <code>has_matcher()</code> returns <code>false</code> and <code>ptr_matcher()</code> returns <code>NULL</code>. See also <a class="el" href="index.html#reflex-multiple-input">Multiple input sources </a>.</p>
<p>The following Flex actions are also supported with <b><code>reflex</code></b> option <code>−−flex</code>:</p>
<table class="doxtable">
<tr>
<th>RE/flex action </th><th>Flex action </th><th>Result  </th></tr>
<tr>
<td><code>in(s)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a3d2545b35aec7c35e6939755777e2a58" title="Flex-compatible macro: scan a string. ">yy_scan_string(s)</a></code> </td><td>reset and scan string <code>s</code> (<code>std::string</code> or <code>char*</code>) </td></tr>
<tr>
<td><code>in(s)</code> </td><td><code><a class="el" href="flexlexer_8h.html#acaf921bfaed6dc25af86fae7ceadc946">yy_scan_wstring(s)</a></code> </td><td>reset and scan wide string <code>s</code> (<code>std::wstring</code> or <code>wchar_t*</code>) </td></tr>
<tr>
<td><code>in(b, n)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a8cef4ba504e54ab2cf998d34ffe047cf">yy_scan_bytes(b, n)</a></code> </td><td>reset and scan <code>n</code> bytes at address <code>b</code> (buffered) </td></tr>
<tr>
<td><code>buffer(b, n+1)</code> </td><td><code>yy_scan_buffer(b, n+2)</code> </td><td>reset and scan <code>n</code> bytes at address <code>b</code> (zero copy) </td></tr>
</table>
<p>These functions create a new buffer (i.e. a new matcher in RE/flex) to incrementally buffer the input on demand, except for <code>yy_scan_buffer</code> that scans a string in place (i.e. zero copy) that should end with two zero bytes, which are included in the specified length. A pointer to the new buffer is returned, which becomes the <code>YY_CURRENT_BUFFER</code>. You should delete the old buffer with <code><a class="el" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229" title="Flex-compatible macro: delete a buffer. ">yy_delete_buffer(YY_CURRENT_BUFFER)</a></code> before creating a new buffer with one of these functions. See <a class="el" href="index.html#reflex-input">Switching input sources </a> for more details.</p>
<p>The generated scanner reads from the standard input by default or from an input source specified as a <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object, such as a string, wide string, file, or a stream. See <a class="el" href="index.html#reflex-input">Switching input sources </a> for more details on managing the input to a scanner.</p>
<p>These functions take an extra last <code>yyscan_t</code> argument for reentrant scanners generated with option <code>−−reentrant</code>. This argument is a pointer to a lexer object. See <a class="el" href="index.html#reflex-reentrant">Reentrant scanners</a> for more details.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-spec-user"></a>
User code sections</h3>
<p>To inject code at the end of the generated scanner, such as a <code>main</code> function, we can use the third and final User code section. All of the code in the User code section is copied to the generated scanner.</p>
<p>Below is a User code section example with <code>main</code> that invokes the lexer to read from standard input (the default input) and display all numbers found:</p>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &lt;iostream&gt;</span></div><div class="line">}</div><div class="line"></div><div class="line">digit       [0-9]</div><div class="line">number      {digit}+</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">{number}    out() &lt;&lt; <span class="stringliteral">&quot;number &quot;</span> &lt;&lt; text() &lt;&lt; std::endl;</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line"><span class="keywordtype">int</span> <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a>() { <span class="keywordflow">return</span> Lexer().lex(); }</div></div><!-- fragment --> </div><p>You can also automatically generate a <code>main</code> with the <b><code>reflex</code></b> <code>−−main</code> option, which will produce the same <code>main</code> function shown in the example above. This creates a stand-alone scanner that instantiates a Lexer that reads input from standard input.</p>
<p>To scan from other input than standard input, such as from files, streams, and strings, instantiate the Lexer class with the input source as the first argument. To set an alternative output stream than standard output, pass a <code>std::ostream</code> object as the second argument to the Lexer class constructor:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="keywordtype">int</span> <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a>(<span class="keywordtype">int</span> argc, <span class="keywordtype">char</span> **argv)</div><div class="line">{</div><div class="line">  FILE *fd = stdin;</div><div class="line">  <span class="keywordflow">if</span> (argc &gt; 1 &amp;&amp; (fd = fopen(argv[1], <span class="stringliteral">&quot;r&quot;</span>)) == NULL)</div><div class="line">    exit(EXIT_FAILURE);</div><div class="line">  std::ofstream of(<span class="stringliteral">&quot;output.txt&quot;</span>, std::ofstream::out);</div><div class="line">  <span class="keywordflow">if</span> (!of)</div><div class="line">    exit(EXIT_FAILURE);</div><div class="line">  Lexer(fd, of).lex();</div><div class="line">  of.close();</div><div class="line">  <span class="keywordflow">if</span> (fd != stdin)</div><div class="line">    fclose(fd);</div><div class="line">  <span class="keywordflow">return</span> 0;</div><div class="line">}</div></div><!-- fragment --> </div><p>The above uses a <code>FILE</code> descriptor to read input from, which has the advantage of automatically decoding UTF-8/16/32 input. Other permissible input sources are <code>std::istream</code>, <code>std::string</code>, <code>std::wstring</code>, <code>char*</code>, and <code>wchar_t*</code>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-patterns"></a>
Patterns                                                     </h2>
<p>The regex pattern syntax you can use generally depends on the regex matcher library that you use. Fortunately, RE/flex accept a broad pattern syntax for lexer specifications. The <b><code>reflex</code></b> command internally converts the regex patterns to regex forms that the underlying matcher engine library can handle (except when specifically indicated in the tables that follow). This ensures that the same pattern syntax can be used with any matcher engine library that RE/flex currently supports.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-syntax"></a>
Pattern syntax</h3>
<p>A pattern is an extended set of regular expressions, with nested sub-expression patterns <code>φ</code> and <code>ψ</code>:</p>
<table class="doxtable">
<tr>
<th>Pattern </th><th>Matches  </th></tr>
<tr>
<td><code>x</code> </td><td>matches the character <code>x</code>, where <code>x</code> is not a special character </td></tr>
<tr>
<td><code>.</code> </td><td>matches any single character except newline (unless in dotall mode) </td></tr>
<tr>
<td><code>\.</code> </td><td>matches <code>.</code> (dot), special characters are escaped with a backslash </td></tr>
<tr>
<td><code>\n</code> </td><td>matches a newline, others are <code>\a</code> (BEL), <code>\b</code> (BS), <code>\t</code> (HT), <code>\v</code> (VT), <code>\f</code> (FF), and <code>\r</code> (CR) </td></tr>
<tr>
<td><code>\N</code> </td><td>matches any single character except newline </td></tr>
<tr>
<td><code>\0</code> </td><td>matches the NUL character </td></tr>
<tr>
<td><code>\cX</code> </td><td>matches the control character <code>X</code> mod 32 (e.g. <code>\cA</code> is <code>\x01</code>) </td></tr>
<tr>
<td><code>\0177</code> </td><td>matches an 8-bit character with octal value <code>177</code> (use <code>\177</code> in lexer specifications instead, see below) </td></tr>
<tr>
<td><code>\x7f</code> </td><td>matches an 8-bit character with hexadecimal value <code>7f</code> </td></tr>
<tr>
<td><code>\x{7f}</code> </td><td>matches an 8-bit character with hexadecimal value <code>7f</code> </td></tr>
<tr>
<td><code>\p{C}</code> </td><td>matches a character in category C of <a class="el" href="index.html#reflex-pattern-cat">Character categories</a> </td></tr>
<tr>
<td><code>\Q...\E</code> </td><td>matches the quoted content between <code>\Q</code> and <code>\E</code> literally </td></tr>
<tr>
<td><code>[abc]</code> </td><td>matches one of <code>a</code>, <code>b</code>, or <code>c</code> as <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
<tr>
<td><code>[0-9]</code> </td><td>matches a digit <code>0</code> to <code>9</code> as <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
<tr>
<td><code>[^0-9]</code> </td><td>matches any character except a digit as <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
<tr>
<td><code>φ?</code> </td><td>matches <code>φ</code> zero or one time (optional) </td></tr>
<tr>
<td><code>φ*</code> </td><td>matches <code>φ</code> zero or more times (repetition) </td></tr>
<tr>
<td><code>φ+</code> </td><td>matches <code>φ</code> one or more times (repetition) </td></tr>
<tr>
<td><code>φ{2,5}</code> </td><td>matches <code>φ</code> two to five times (repetition) </td></tr>
<tr>
<td><code>φ{2,}</code> </td><td>matches <code>φ</code> at least two times (repetition) </td></tr>
<tr>
<td><code>φ{2}</code> </td><td>matches <code>φ</code> exactly two times (repetition) </td></tr>
<tr>
<td><code>φ??</code> </td><td>matches <code>φ</code> zero or once as needed (lazy optional) </td></tr>
<tr>
<td><code>φ*?</code> </td><td>matches <code>φ</code> a minimum number of times as needed (lazy repetition) </td></tr>
<tr>
<td><code>φ+?</code> </td><td>matches <code>φ</code> a minimum number of times at least once as needed (lazy repetition) </td></tr>
<tr>
<td><code>φ{2,5}?</code> </td><td>matches <code>φ</code> two to five times as needed (lazy repetition) </td></tr>
<tr>
<td><code>φ{2,}?</code> </td><td>matches <code>φ</code> at least two times or more as needed (lazy repetition) </td></tr>
<tr>
<td><code>φψ</code> </td><td>matches <code>φ</code> then matches <code>ψ</code> (concatenation) </td></tr>
<tr>
<td><code>φ⎮ψ</code> </td><td>matches <code>φ</code> or matches <code>ψ</code> (alternation) </td></tr>
<tr>
<td><code>(φ)</code> </td><td>matches <code>φ</code> as a group to capture (this is non-capturing in lexer specifications) </td></tr>
<tr>
<td><code>(?:φ)</code> </td><td>matches <code>φ</code> without group capture </td></tr>
<tr>
<td><code>(?=φ)</code> </td><td>matches <code>φ</code> without consuming it (<a class="el" href="index.html#reflex-pattern-lookahead">Lookahead</a>) </td></tr>
<tr>
<td><code>(?&lt;=φ)</code> </td><td>matches <code>φ</code> to the left without consuming it (<a class="el" href="index.html#reflex-pattern-lookbehind">Lookbehind</a>, not supported by the RE/flex matcher) </td></tr>
<tr>
<td><code>(?^φ)</code> </td><td>matches <code>φ</code> and ignore it to continue matching (RE/flex matcher only) </td></tr>
<tr>
<td><code>^φ</code> </td><td>matches <code>φ</code> at the begin of input or begin of a line (requires multi-line mode) (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>φ$</code> </td><td>matches <code>φ</code> at the end of input or end of a line (requires multi-line mode) (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>\Aφ</code> </td><td>matches <code>φ</code> at the begin of input (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>φ\z</code> </td><td>matches <code>φ</code> at the end of input (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>\bφ</code> </td><td>matches <code>φ</code> starting at a word boundary (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>φ\b</code> </td><td>matches <code>φ</code> ending at a word boundary (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>\Bφ</code> </td><td>matches <code>φ</code> starting at a non-word boundary (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>φ\B</code> </td><td>matches <code>φ</code> ending at a non-word boundary (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>\&lt;φ</code> </td><td>matches <code>φ</code> that starts a word (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>\&gt;φ</code> </td><td>matches <code>φ</code> that starts a non-word (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>φ\&lt;</code> </td><td>matches <code>φ</code> that ends a non-word (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>φ\&gt;</code> </td><td>matches <code>φ</code> that ends a word (top-level <code>φ</code>, not nested in a sub-pattern) </td></tr>
<tr>
<td><code>\i</code> </td><td>matches an indent for <a class="el" href="index.html#reflex-pattern-dents">Indent/nodent/dedent</a> matching </td></tr>
<tr>
<td><code>\j</code> </td><td>matches a dedent for <a class="el" href="index.html#reflex-pattern-dents">Indent/nodent/dedent</a> matching </td></tr>
<tr>
<td><code>\k</code> </td><td>matches if indent depth changed, undoing this change to keep the current indent stops for <a class="el" href="index.html#reflex-pattern-dents">Indent/nodent/dedent</a> matching </td></tr>
<tr>
<td><code>(?i:φ)</code> </td><td><a class="el" href="index.html#reflex-pattern-anycase">Case-insensitive mode</a> matches <code>φ</code> ignoring case </td></tr>
<tr>
<td><code>(?m:φ)</code> </td><td><a class="el" href="index.html#reflex-pattern-multiline">Multi-line mode</a> <code>^</code> and <code>$</code> in <code>φ</code> match begin and end of a line (default in lexer specifications) </td></tr>
<tr>
<td><code>(?s:φ)</code> </td><td><a class="el" href="index.html#reflex-pattern-dotall">Dotall mode</a> <code>.</code> (dot) in <code>φ</code> matches newline </td></tr>
<tr>
<td><code>(?u:φ)</code> </td><td><a class="el" href="index.html#reflex-pattern-unicode">Unicode mode</a> <code>.</code>, <code>\s</code>, <code>\w</code>, <code>\l</code>, <code>\u</code>, <code>\S</code>, <code>\W</code>, <code>\L</code>, <code>\U</code> match Unicode </td></tr>
<tr>
<td><code>(?x:φ)</code> </td><td><a class="el" href="index.html#reflex-pattern-freespace">Free space mode</a> ignore all whitespace and comments in <code>φ</code> </td></tr>
<tr>
<td><code>(?#:X)</code> </td><td>all of <code>X</code> is skipped as a comment </td></tr>
</table>
<dl class="section note"><dt>Note</dt><dd>The lazy quantifier <code>?</code> for optional patterns <code>φ??</code> and repetitions <code>φ*?</code> <code>φ+?</code> is not supported by Boost.Regex in POSIX mode. In general, POSIX matchers do not support lazy quantifiers due to POSIX limitations that are rooted in the theory of formal languages FSM of regular expressions.</dd></dl>
<p>The following patterns are available in RE/flex and adopt the same Flex/Lex patterns syntax. These pattern should only be used in lexer specifications:</p>
<table class="doxtable">
<tr>
<th>Pattern </th><th>Matches  </th></tr>
<tr>
<td><code>\177</code> </td><td>matches an 8-bit character with octal value <code>177</code> </td></tr>
<tr>
<td><code>"..."</code> </td><td>matches the quoted content literally </td></tr>
<tr>
<td><code>φ/ψ</code> </td><td>matches <code>φ</code> if followed by <code>ψ</code> as a <a class="el" href="index.html#reflex-pattern-trailing">Trailing context</a> </td></tr>
<tr>
<td><code>&lt;S&gt;φ</code> </td><td>matches <code>φ</code> only if state <code>S</code> is enabled in <a class="el" href="index.html#reflex-states">Start condition states </a> </td></tr>
<tr>
<td><code>&lt;S1,S2,S3&gt;φ</code> </td><td>matches <code>φ</code> only if state <code>S1</code>, <code>S2</code>, or state <code>S3</code> is enabled in <a class="el" href="index.html#reflex-states">Start condition states </a> </td></tr>
<tr>
<td><code>&lt;*&gt;φ</code> </td><td>matches <code>φ</code> in any state of the <a class="el" href="index.html#reflex-states">Start condition states </a> </td></tr>
<tr>
<td><code>&lt;&lt;EOF&gt;&gt;</code> </td><td>matches EOF in any state of the <a class="el" href="index.html#reflex-states">Start condition states </a> </td></tr>
<tr>
<td><code>&lt;S&gt;&lt;&lt;EOF&gt;&gt;</code> </td><td>matches EOF only if state <code>S</code> is enabled in <a class="el" href="index.html#reflex-states">Start condition states </a> </td></tr>
<tr>
<td><code>[a-z⎮⎮[A-Z]]</code> </td><td>matches a letter, see <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
<tr>
<td><code>[a-z&amp;&amp;[^aeiou]]</code> </td><td>matches a consonant, see <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
<tr>
<td><code>[a-z−−[aeiou]]</code> </td><td>matches a consonant, see <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
<tr>
<td><code>[a-z]{+}[A-Z]</code> </td><td>matches a letter, same as <code>[a-z⎮⎮[A-Z]]</code>, see <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
<tr>
<td><code>[a-z]{⎮}[A-Z]</code> </td><td>matches a letter, same as <code>[a-z⎮⎮[A-Z]]</code>, see <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
<tr>
<td><code>[a-z]{&amp;}[^aeiou]</code> </td><td>matches a consonant, same as <code>[a-z&amp;&amp;[^aeiou]]</code>, see <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
<tr>
<td><code>[a-z]{-}[aeiou]</code> </td><td>matches a consonant, same as <code>[a-z−−[aeiou]]</code>, see <a class="el" href="index.html#reflex-pattern-class">Character classes</a> </td></tr>
</table>
<p>Note that the characters <code>.</code> (dot), <code>\</code>, <code>?</code>, <code>*</code>, <code>+</code>, <code>|</code>, <code>(</code>, <code>)</code>, <code>[</code>, <code>]</code>, <code>{</code>, <code>}</code>, <code>^</code>, and <code>$</code> are meta-characters and should be escaped to match. Lexer specifications also include the <code>"</code> and <code>/</code> as meta-characters and these should be escaped to match.</p>
<p>Spaces and tabs cannot be matched in patterns in lexer specifications. To match the space character use <code>" "</code> or <code>[ ]</code> and to match the tab character use <code>\t</code>. Use <code>\h</code> to match a space or tab.</p>
<p>The order of precedence for composing larger patterns from sub-patterns is as follows, from high to low precedence:</p>
<ol type="1">
<li>Characters, character classes (bracket expressions), escapes, quotation</li>
<li>Grouping <code>(φ)</code>, <code>(?:φ)</code>, <code>(?=φ)</code>, and inline modifiers <code>(?imsux-imsux:φ)</code></li>
<li>Quantifiers <code>?</code>, <code>*</code>, <code>+</code>, <code>{n,m}</code></li>
<li>Concatenation <code>φψ</code> (including trailing context <code>φ/ψ</code>)</li>
<li>Anchoring <code>^</code>, <code>$</code>, <code>\&lt;</code>, <code>\&gt;</code>, <code>\b</code>, <code>\B</code>, <code>\A</code>, <code>\z</code></li>
<li>Alternation <code>φ|ψ</code></li>
<li>Global modifiers <code>(?imsux-imsux)φ</code></li>
</ol>
<dl class="section note"><dt>Note</dt><dd>When using regex patterns in C++ literal strings, make sure that "regex
escapes are escaped", meaning that an extra backslash is needed for every backslash in the regex string.</dd>
<dd>
Trigraphs in C/C++ strings are special tripple-character sequences, beginning with two question marks and followed by a character that is translated. Avoid <code>??</code> in regex strings. Instead, use at least one escaped question mark, such as <code>?\?</code>, which the compiler will translate to <code>??</code>. This problem does not apply to lexer specifications that the <b><code>reflex</code></b> command converts to regex strings. Fortunately, most C++ compilers ignore trigraphs unless in standard-conforming modes, such as <code>-ansi</code> and <code>-std=c++98</code>.</dd></dl>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-class"></a>
Character classes</h3>
<p>Character classes in bracket lists represent sets of characters. Sets can be negated (or inverted), subtracted, intersected, and merged:</p>
<table class="doxtable">
<tr>
<th>Pattern </th><th>Matches  </th></tr>
<tr>
<td><code>[a-zA-Z]</code> </td><td>matches a letter </td></tr>
<tr>
<td><code>[^a-zA-Z]</code> </td><td>matches a non-letter (character class negation) </td></tr>
<tr>
<td><code>[a-z⎮⎮[A-Z]]</code> </td><td>matches a letter (character class union) </td></tr>
<tr>
<td><code>[a-z&amp;&amp;[^aeiou]]</code> </td><td>matches a consonant (character class intersection) </td></tr>
<tr>
<td><code>[a-z−−[aeiou]]</code> </td><td>matches a consonant (character class subtraction) </td></tr>
</table>
<p>Bracket lists cannot be empty, so <code>[]</code> and <code>[^]</code> are invalid. In fact, the first character after the bracket is always part of the list. So <code>[][]</code> is a list that matches a <code>]</code> and a <code>[</code>, <code>[^][]</code> is a list that matches anything but <code>]</code> and <code>[</code>, and <code>[-^]</code> is a list that matches a <code>-</code> and a <code>^</code>.</p>
<p>It is an error to construct an empty character class by subtraction or by intersection, for example <code>[a&amp;&amp;[b]]</code> is invalid.</p>
<p>Bracket lists may contain ASCII and Unicode <a class="el" href="index.html#reflex-pattern-cat">Character categories</a>, for example <code>[a-z\d]</code> and <code>[a-z[:digit:]]</code> contain the letters <code>a</code> to <code>z</code> and digits <code>0</code> to <code>9</code>. To add Unicode character categories and wide characters (encoded in UTF-8) to bracket lists <a class="el" href="index.html#reflex-pattern-unicode">Unicode mode</a> should be enabled.</p>
<p>An negated Unicode character class is constructed by subtracting the character class from the Unicode range U+0000 to U+D7FF and U+E000 to U+10FFFF.</p>
<p>Character class operations can be chained together in a bracket list. The union <code>||</code>, intersection <code>&amp;&amp;</code>, and subtraction <code>--</code> operations are left associative and have the same operator precedence. For example, <code>[a-z||[A-Z]--[aeiou]--[AEIOU]]</code>, <code>[a-z--[aeiou]||[A-Z]--[AEIUO]]</code>, <code>[a-z&amp;&amp;[^aeiou]||[A-Z]&amp;&amp;[^AEIOU]]</code>, and <code>[B-DF-HJ-NP-TV-Zb-df-hj-np-tv-z]</code> are the same character classes.</p>
<p>Character class operations may be nested. For example, <code>[a-z||[A-Z||[0-9]]]</code> is the same as <code>[a-zA-Z0-9]</code>.</p>
<p>Character class negation, when specified, is applied to the resulting character class after the character class operations are applied. For example, <code>[^a-z||[A-Z]]</code> is the same as <code>[^||[a-z]||[A-Z]]</code>, which is the class <code>[^a-zA-Z]</code>.</p>
<p>Note that negated character classes such as <code>[^a-zA-Z]</code> match newlines when <code>\n</code> is not included in the class. Include <code>\n</code> in the negated character class to prevent matching newlines. The <code><a class="el" href="namespacereflex_1_1convert__flag.html#aaed1f496284028394303d24be3e7bbb9" title="inverted character classes do not match newline \n ">reflex::convert_flag::notnewline</a></code> removes newlines from negated character classes when used with <a class="el" href="index.html#regex-convert">Regex converters </a>.</p>
<p>A lexer specification may use a defined name in place of the second operand of a character class operation. A defined name when used as an operand should expand into a POSIX character class containing ASCII characters only. For example:</p>
<div class="alt"> <div class="fragment"><div class="line"><a class="code" href="reflex_8cpp.html#afd21dbba9aaf2944d477caa4caddd963">lower</a>     [a-z]</div><div class="line">upper     [A-Z]</div><div class="line">letter    [||{<a class="code" href="reflex_8cpp.html#afd21dbba9aaf2944d477caa4caddd963">lower</a>}||{upper}]</div><div class="line">alnum     [0-9||{letter}]</div><div class="line">name      {letter}{alnum}*</div><div class="line">%%</div><div class="line">{name}    std::cout &lt;&lt; <span class="stringliteral">&quot;name: &quot;</span> &lt;&lt; text() &lt;&lt; std::endl;</div><div class="line">.|\n      <span class="comment">// skip everything else</span></div><div class="line">%%</div></div><!-- fragment --> </div><dl class="section warning"><dt>Warning</dt><dd>Defined names may only occur immediately after a <code>||</code>, <code>&amp;&amp;</code>, and a <code>--</code> operator in a bracket list. Do not place a defined name as the first operand to a union, intersection, and subtraction operation, because the definition is not expanded. For example, <code>[{lower}||{upper}]</code> contains <code>[A-Zelorw{}]</code>. The name and the <code>{</code>, <code>}</code> characters are literally included in the resulting character class. Instead, this bracket list should be written as <code>[||{lower}||{upper}]</code>. Likewise, <code>[^{lower}||{upper}]</code> should be written as <code>[^||{lower}||{upper}]</code>.</dd></dl>
<p>Alternatively, unions may be written as alternations. That is, <code>[||{name1}||{name2}||{name3}||...]</code> can be written as <code>({name1}|{name2}|{name3}|...)</code>, where the latter form supports full Unicode not restricted to ASCII.</p>
<p>The character class operators <code>{+}</code> (or <code>{|}</code>), <code>{&amp;}</code>, and <code>{-}</code> may be used in lexer specifications. Note that Flex only supports the two operators <code>{+}</code> and <code>{-}</code>:</p>
<table class="doxtable">
<tr>
<th>Pattern </th><th>Matches  </th></tr>
<tr>
<td><code>[a-z]{+}[A-Z]</code> </td><td>matches a letter, same as <code>[a-z⎮⎮[A-Z]]</code> </td></tr>
<tr>
<td><code>[a-z]{⎮}[A-Z]</code> </td><td>matches a letter, same as <code>[a-z⎮⎮[A-Z]]</code> </td></tr>
<tr>
<td><code>[a-z]{&amp;}[^aeiou]</code> </td><td>matches a consonant, same as <code>[a-z&amp;&amp;[^aeiou]]</code> </td></tr>
<tr>
<td><code>[a-z]{-}[aeiou]</code> </td><td>matches a consonant, same as <code>[a-z−−[aeiou]]</code> </td></tr>
</table>
<p>Multiple operators can be chained together. Unlike Flex, defined names may be used as operands. For example <code>{lower}{+}{upper}</code> is the same as <code>[a-z]{+}[A-Z]</code>, i.e. the character class <code>[A-Za-z]</code>. A defined name when used as an operand should expand into a POSIX character class containing ASCII characters only.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-cat"></a>
Character categories</h3>
<p>The 7-bit ASCII POSIX character categories are:</p>
<table class="doxtable">
<tr>
<th>POSIX form </th><th>POSIX category </th><th>Matches  </th></tr>
<tr>
<td><code>[:ascii:]</code> </td><td><code>\p{ASCII}</code> </td><td>matches any ASCII character </td></tr>
<tr>
<td><code>[:space:]</code> </td><td><code>\p{Space}</code> </td><td>matches a white space character <code>[ \t\n\v\f\r]</code> </td></tr>
<tr>
<td><code>[:xdigit:]</code> </td><td><code>\p{Xdigit}</code> </td><td>matches a hex digit <code>[0-9A-Fa-f]</code> </td></tr>
<tr>
<td><code>[:cntrl:]</code> </td><td><code>\p{Cntrl}</code> </td><td>matches a control character <code>[\x00-\0x1f\x7f]</code> </td></tr>
<tr>
<td><code>[:print:]</code> </td><td><code>\p{Print}</code> </td><td>matches a printable character <code>[\x20-\x7e]</code> </td></tr>
<tr>
<td><code>[:alnum:]</code> </td><td><code>\p{Alnum}</code> </td><td>matches a alphanumeric character <code>[0-9A-Za-z]</code> </td></tr>
<tr>
<td><code>[:alpha:]</code> </td><td><code>\p{Alpha}</code> </td><td>matches a letter <code>[A-Za-z]</code> </td></tr>
<tr>
<td><code>[:blank:]</code> </td><td><code>\p{Blank}</code>, <code>\h</code> </td><td>matches a blank <code>[ \t]</code> </td></tr>
<tr>
<td><code>[:digit:]</code> </td><td><code>\p{Digit}</code>, <code>\d</code> </td><td>matches a digit <code>[0-9]</code> </td></tr>
<tr>
<td><code>[:graph:]</code> </td><td><code>\p{Graph}</code> </td><td>matches a visible character <code>[\x21-\x7e]</code> </td></tr>
<tr>
<td><code>[:lower:]</code> </td><td><code>\p{Lower}</code> </td><td>matches a lower case letter <code>[a-z]</code> </td></tr>
<tr>
<td><code>[:punct:]</code> </td><td><code>\p{Punct}</code> </td><td>matches a punctuation character <code>[\x21-\x2f\x3a-\x40\x5b-\x60\x7b-\x7e]</code> </td></tr>
<tr>
<td><code>[:upper:]</code> </td><td><code>\p{Upper}</code> </td><td>matches an upper case letter <code>[A-Z]</code> </td></tr>
<tr>
<td><code>[:word:]</code> </td><td><code>\p{Word}</code> </td><td>matches a word character <code>[0-9A-Za-z_]</code> </td></tr>
<tr>
<td><code>[:^blank:]</code> </td><td><code>\P{Blank}</code>, <code>\H</code> </td><td>matches a non-blank character <code>[^ \t]</code> </td></tr>
<tr>
<td><code>[:^digit:]</code> </td><td><code>\P{Digit}</code>, <code>\D</code> </td><td>matches a non-digit <code>[^0-9]</code> </td></tr>
</table>
<p>The POSIX forms are used in bracket lists. For example <code>[[:lower:][:digit:]]</code> matches an ASCII lower case letter or a digit.</p>
<p>You can also use the upper case <code>\P{C}</code> form that has the same meaning as <code>\p{^C}</code>, which matches any character except characters in the class <code>C</code>. For example, <code>\P{ASCII}</code> is the same as <code>\p{^ASCII}</code> which is the same as <code>[^[:ascii:]]</code>.</p>
<p>When Unicode matching mode is enabled, <code>[^[:ascii]]</code> is a Unicode character class that excludes the ASCII character category. Unicode character classes and categories require the <b><code>reflex</code></b> <code>−−unicode</code> option.</p>
<p>The following Unicode character categories are enabled with the <b><code>reflex</code></b> <code>−−unicode</code> option or <a class="el" href="index.html#reflex-pattern-unicode">Unicode mode</a> <code>(?u:φ)</code> and with the regex matcher converter flag <code><a class="el" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39" title="convert . (dot), \s, \w, \l, \u, \S, \W, \L, \U to Unicode ">reflex::convert_flag::unicode</a></code> when using a regex library:</p>
<table class="doxtable">
<tr>
<th>Unicode category </th><th>Matches  </th></tr>
<tr>
<td><code>.</code> </td><td>matches any single Unicode character except newline (including <a class="el" href="index.html#invalid-utf">Invalid UTF encodings </a>) </td></tr>
<tr>
<td><code>\X</code> </td><td>matches any Unicode character (with or without the <code>−−unicode</code> option) </td></tr>
<tr>
<td><code>\x{3B1}</code>, <code>\u{3B1}</code> </td><td>matches Unicode character U+03B1, i.e. <code>α</code> </td></tr>
<tr>
<td><code>\R</code> </td><td>matches a Unicode line break </td></tr>
<tr>
<td><code>\s</code>, <code>\p{Zs}</code> </td><td>matches a white space character with Unicode sub-propert Zs </td></tr>
<tr>
<td><code>\l</code>, <code>\p{Ll}</code> </td><td>matches a lower case letter with Unicode sub-property Ll </td></tr>
<tr>
<td><code>\u</code>, <code>\p{Lu}</code> </td><td>matches an upper case letter with Unicode sub-property Lu </td></tr>
<tr>
<td><code>\w</code>, <code>\p{Word}</code> </td><td>matches a Unicode word character with property L, Nd, or Pc </td></tr>
<tr>
<td><code>\p{Unicode}</code> </td><td>matches any Unicode character (U+0000 to U+10FFFF minus U+D800 to U+DFFF) </td></tr>
<tr>
<td><code>\p{ASCII}</code> </td><td>matches an ASCII character U+0000 to U+007F) </td></tr>
<tr>
<td><code>\p{Non_ASCII_Unicode}</code> </td><td>matches a non-ASCII character U+0080 to U+10FFFF minus U+D800 to U+DFFF) </td></tr>
<tr>
<td><code>\p{L&amp;}</code> </td><td>matches a character with Unicode property L&amp; (i.e. property Ll, Lu, or Lt) </td></tr>
<tr>
<td><code>\p{Letter}</code>,<code>\p{L}</code> </td><td>matches a character with Unicode property Letter </td></tr>
<tr>
<td><code>\p{Mark}</code>,<code>\p{M}</code> </td><td>matches a character with Unicode property Mark </td></tr>
<tr>
<td><code>\p{Separator}</code>,<code>\p{Z}</code> </td><td>matches a character with Unicode property Separator </td></tr>
<tr>
<td><code>\p{Symbol}</code>,<code>\p{S}</code> </td><td>matches a character with Unicode property Symbol </td></tr>
<tr>
<td><code>\p{Number}</code>,<code>\p{N}</code> </td><td>matches a character with Unicode property Number </td></tr>
<tr>
<td><code>\p{Punctuation}</code>,<code>\p{P}</code> </td><td>matches a character with Unicode property Punctuation </td></tr>
<tr>
<td><code>\p{Other}</code>,<code>\p{C}</code> </td><td>matches a character with Unicode property Other </td></tr>
<tr>
<td><code>\p{Lowercase_Letter}</code>, <code>\p{Ll}</code> </td><td>matches a character with Unicode sub-property Ll </td></tr>
<tr>
<td><code>\p{Uppercase_Letter}</code>, <code>\p{Lu}</code> </td><td>matches a character with Unicode sub-property Lu </td></tr>
<tr>
<td><code>\p{Titlecase_Letter}</code>, <code>\p{Lt}</code> </td><td>matches a character with Unicode sub-property Lt </td></tr>
<tr>
<td><code>\p{Modifier_Letter}</code>, <code>\p{Lm}</code> </td><td>matches a character with Unicode sub-property Lm </td></tr>
<tr>
<td><code>\p{Other_Letter}</code>, <code>\p{Lo}</code> </td><td>matches a character with Unicode sub-property Lo </td></tr>
<tr>
<td><code>\p{Non_Spacing_Mark}</code>, <code>\p{Mn}</code> </td><td>matches a character with Unicode sub-property Mn </td></tr>
<tr>
<td><code>\p{Spacing_Combining_Mark}</code>, <code>\p{Mc}</code> </td><td>matches a character with Unicode sub-property Mc </td></tr>
<tr>
<td><code>\p{Enclosing_Mark}</code>, <code>\p{Me}</code> </td><td>matches a character with Unicode sub-property Me </td></tr>
<tr>
<td><code>\p{Space_Separator}</code>, <code>\p{Zs}</code> </td><td>matches a character with Unicode sub-property Zs </td></tr>
<tr>
<td><code>\p{Line_Separator}</code>, <code>\p{Zl}</code> </td><td>matches a character with Unicode sub-property Zl </td></tr>
<tr>
<td><code>\p{Paragraph_Separator}</code>, <code>\p{Zp}</code> </td><td>matches a character with Unicode sub-property Zp </td></tr>
<tr>
<td><code>\p{Math_Symbol}</code>, <code>\p{Sm}</code> </td><td>matches a character with Unicode sub-property Sm </td></tr>
<tr>
<td><code>\p{Currency_Symbol}</code>, <code>\p{Sc}</code> </td><td>matches a character with Unicode sub-property Sc </td></tr>
<tr>
<td><code>\p{Modifier_Symbol}</code>, <code>\p{Sk}</code> </td><td>matches a character with Unicode sub-property Sk </td></tr>
<tr>
<td><code>\p{Other_Symbol}</code>, <code>\p{So}</code> </td><td>matches a character with Unicode sub-property So </td></tr>
<tr>
<td><code>\p{Decimal_Digit_Number}</code>, <code>\p{Nd}</code> </td><td>matches a character with Unicode sub-property Nd </td></tr>
<tr>
<td><code>\p{Letter_Number}</code>, <code>\p{Nl}</code> </td><td>matches a character with Unicode sub-property Nl </td></tr>
<tr>
<td><code>\p{Other_Number}</code>, <code>\p{No}</code> </td><td>matches a character with Unicode sub-property No </td></tr>
<tr>
<td><code>\p{Dash_Punctuation}</code>, <code>\p{Pd}</code> </td><td>matches a character with Unicode sub-property Pd </td></tr>
<tr>
<td><code>\p{Open_Punctuation}</code>, <code>\p{Ps}</code> </td><td>matches a character with Unicode sub-property Ps </td></tr>
<tr>
<td><code>\p{Close_Punctuation}</code>, <code>\p{Pe}</code> </td><td>matches a character with Unicode sub-property Pe </td></tr>
<tr>
<td><code>\p{Initial_Punctuation}</code>, <code>\p{Pi}</code> </td><td>matches a character with Unicode sub-property Pi </td></tr>
<tr>
<td><code>\p{Final_Punctuation}</code>, <code>\p{Pf}</code> </td><td>matches a character with Unicode sub-property Pf </td></tr>
<tr>
<td><code>\p{Connector_Punctuation}</code>, <code>\p{Pc}</code> </td><td>matches a character with Unicode sub-property Pc </td></tr>
<tr>
<td><code>\p{Other_Punctuation}</code>, <code>\p{Po}</code> </td><td>matches a character with Unicode sub-property Po </td></tr>
<tr>
<td><code>\p{Control}</code>, <code>\p{Cc}</code> </td><td>matches a character with Unicode sub-property Cc </td></tr>
<tr>
<td><code>\p{Format}</code>, <code>\p{Cf}</code> </td><td>matches a character with Unicode sub-property Cf </td></tr>
<tr>
<td><code>\p{UnicodeIdentifierStart}</code> </td><td>matches a character in the Unicode IdentifierStart class </td></tr>
<tr>
<td><code>\p{UnicodeIdentifierPart}</code> </td><td>matches a character in the Unicode IdentifierPart class </td></tr>
<tr>
<td><code>\p{IdentifierIgnorable}</code> </td><td>matches a character in the IdentifierIgnorable class </td></tr>
<tr>
<td><code>\p{JavaIdentifierStart}</code> </td><td>matches a character in the Java IdentifierStart class </td></tr>
<tr>
<td><code>\p{JavaIdentifierPart}</code> </td><td>matches a character in the Java IdentifierPart class </td></tr>
<tr>
<td><code>\p{CsIdentifierStart}</code> </td><td>matches a character in the C# IdentifierStart class </td></tr>
<tr>
<td><code>\p{CsIdentifierPart}</code> </td><td>matches a character in the C# IdentifierPart class </td></tr>
<tr>
<td><code>\p{PythonIdentifierStart}</code> </td><td>matches a character in the Python IdentifierStart class </td></tr>
<tr>
<td><code>\p{PythonIdentifierPart}</code> </td><td>matches a character in the Python IdentifierPart class </td></tr>
</table>
<p>To specify a Unicode block as a category when using the <code>−−unicode</code> option, use <code>\p{IsBlockName}</code>. The table below lists the block categories up to U+FFFF, However, all Unicode blocks up to U+10FFFF are available (not listed):</p>
<table class="doxtable">
<tr>
<th>IsBlockName </th><th>Unicode character range  </th></tr>
<tr>
<td><code>\p{IsBasicLatin}</code> </td><td>U+0000 to U+007F </td></tr>
<tr>
<td><code>\p{IsLatin-1Supplement}</code> </td><td>U+0080 to U+00FF </td></tr>
<tr>
<td><code>\p{IsLatinExtended-A}</code> </td><td>U+0100 to U+017F </td></tr>
<tr>
<td><code>\p{IsLatinExtended-B}</code> </td><td>U+0180 to U+024F </td></tr>
<tr>
<td><code>\p{IsIPAExtensions}</code> </td><td>U+0250 to U+02AF </td></tr>
<tr>
<td><code>\p{IsSpacingModifierLetters}</code> </td><td>U+02B0 to U+02FF </td></tr>
<tr>
<td><code>\p{IsCombiningDiacriticalMarks}</code> </td><td>U+0300 to U+036F </td></tr>
<tr>
<td><code>\p{IsGreekandCoptic}</code> </td><td>U+0370 to U+03FF </td></tr>
<tr>
<td><code>\p{IsCyrillic}</code> </td><td>U+0400 to U+04FF </td></tr>
<tr>
<td><code>\p{IsCyrillicSupplement}</code> </td><td>U+0500 to U+052F </td></tr>
<tr>
<td><code>\p{IsArmenian}</code> </td><td>U+0530 to U+058F </td></tr>
<tr>
<td><code>\p{IsHebrew}</code> </td><td>U+0590 to U+05FF </td></tr>
<tr>
<td><code>\p{IsArabic}</code> </td><td>U+0600 to U+06FF </td></tr>
<tr>
<td><code>\p{IsSyriac}</code> </td><td>U+0700 to U+074F </td></tr>
<tr>
<td><code>\p{IsArabicSupplement}</code> </td><td>U+0750 to U+077F </td></tr>
<tr>
<td><code>\p{IsThaana}</code> </td><td>U+0780 to U+07BF </td></tr>
<tr>
<td><code>\p{IsNKo}</code> </td><td>U+07C0 to U+07FF </td></tr>
<tr>
<td><code>\p{IsSamaritan}</code> </td><td>U+0800 to U+083F </td></tr>
<tr>
<td><code>\p{IsMandaic}</code> </td><td>U+0840 to U+085F </td></tr>
<tr>
<td><code>\p{IsSyriacSupplement}</code> </td><td>U+0860 to U+086F </td></tr>
<tr>
<td><code>\p{IsArabicExtended-A}</code> </td><td>U+08A0 to U+08FF </td></tr>
<tr>
<td><code>\p{IsDevanagari}</code> </td><td>U+0900 to U+097F </td></tr>
<tr>
<td><code>\p{IsBengali}</code> </td><td>U+0980 to U+09FF </td></tr>
<tr>
<td><code>\p{IsGurmukhi}</code> </td><td>U+0A00 to U+0A7F </td></tr>
<tr>
<td><code>\p{IsGujarati}</code> </td><td>U+0A80 to U+0AFF </td></tr>
<tr>
<td><code>\p{IsOriya}</code> </td><td>U+0B00 to U+0B7F </td></tr>
<tr>
<td><code>\p{IsTamil}</code> </td><td>U+0B80 to U+0BFF </td></tr>
<tr>
<td><code>\p{IsTelugu}</code> </td><td>U+0C00 to U+0C7F </td></tr>
<tr>
<td><code>\p{IsKannada}</code> </td><td>U+0C80 to U+0CFF </td></tr>
<tr>
<td><code>\p{IsMalayalam}</code> </td><td>U+0D00 to U+0D7F </td></tr>
<tr>
<td><code>\p{IsSinhala}</code> </td><td>U+0D80 to U+0DFF </td></tr>
<tr>
<td><code>\p{IsThai}</code> </td><td>U+0E00 to U+0E7F </td></tr>
<tr>
<td><code>\p{IsLao}</code> </td><td>U+0E80 to U+0EFF </td></tr>
<tr>
<td><code>\p{IsTibetan}</code> </td><td>U+0F00 to U+0FFF </td></tr>
<tr>
<td><code>\p{IsMyanmar}</code> </td><td>U+1000 to U+109F </td></tr>
<tr>
<td><code>\p{IsGeorgian}</code> </td><td>U+10A0 to U+10FF </td></tr>
<tr>
<td><code>\p{IsHangulJamo}</code> </td><td>U+1100 to U+11FF </td></tr>
<tr>
<td><code>\p{IsEthiopic}</code> </td><td>U+1200 to U+137F </td></tr>
<tr>
<td><code>\p{IsEthiopicSupplement}</code> </td><td>U+1380 to U+139F </td></tr>
<tr>
<td><code>\p{IsCherokee}</code> </td><td>U+13A0 to U+13FF </td></tr>
<tr>
<td><code>\p{IsUnifiedCanadianAboriginalSyllabics}</code> </td><td>U+1400 to U+167F </td></tr>
<tr>
<td><code>\p{IsOgham}</code> </td><td>U+1680 to U+169F </td></tr>
<tr>
<td><code>\p{IsRunic}</code> </td><td>U+16A0 to U+16FF </td></tr>
<tr>
<td><code>\p{IsTagalog}</code> </td><td>U+1700 to U+171F </td></tr>
<tr>
<td><code>\p{IsHanunoo}</code> </td><td>U+1720 to U+173F </td></tr>
<tr>
<td><code>\p{IsBuhid}</code> </td><td>U+1740 to U+175F </td></tr>
<tr>
<td><code>\p{IsTagbanwa}</code> </td><td>U+1760 to U+177F </td></tr>
<tr>
<td><code>\p{IsKhmer}</code> </td><td>U+1780 to U+17FF </td></tr>
<tr>
<td><code>\p{IsMongolian}</code> </td><td>U+1800 to U+18AF </td></tr>
<tr>
<td><code>\p{IsUnifiedCanadianAboriginalSyllabicsExtended}</code> </td><td>U+18B0 to U+18FF </td></tr>
<tr>
<td><code>\p{IsLimbu}</code> </td><td>U+1900 to U+194F </td></tr>
<tr>
<td><code>\p{IsTaiLe}</code> </td><td>U+1950 to U+197F </td></tr>
<tr>
<td><code>\p{IsNewTaiLue}</code> </td><td>U+1980 to U+19DF </td></tr>
<tr>
<td><code>\p{IsKhmerSymbols}</code> </td><td>U+19E0 to U+19FF </td></tr>
<tr>
<td><code>\p{IsBuginese}</code> </td><td>U+1A00 to U+1A1F </td></tr>
<tr>
<td><code>\p{IsTaiTham}</code> </td><td>U+1A20 to U+1AAF </td></tr>
<tr>
<td><code>\p{IsCombiningDiacriticalMarksExtended}</code> </td><td>U+1AB0 to U+1AFF </td></tr>
<tr>
<td><code>\p{IsBalinese}</code> </td><td>U+1B00 to U+1B7F </td></tr>
<tr>
<td><code>\p{IsSundanese}</code> </td><td>U+1B80 to U+1BBF </td></tr>
<tr>
<td><code>\p{IsBatak}</code> </td><td>U+1BC0 to U+1BFF </td></tr>
<tr>
<td><code>\p{IsLepcha}</code> </td><td>U+1C00 to U+1C4F </td></tr>
<tr>
<td><code>\p{IsOlChiki}</code> </td><td>U+1C50 to U+1C7F </td></tr>
<tr>
<td><code>\p{IsCyrillicExtended-C}</code> </td><td>U+1C80 to U+1C8F </td></tr>
<tr>
<td><code>\p{IsGeorgianExtended}</code> </td><td>U+1C90 to U+1CBF </td></tr>
<tr>
<td><code>\p{IsSundaneseSupplement}</code> </td><td>U+1CC0 to U+1CCF </td></tr>
<tr>
<td><code>\p{IsVedicExtensions}</code> </td><td>U+1CD0 to U+1CFF </td></tr>
<tr>
<td><code>\p{IsPhoneticExtensions}</code> </td><td>U+1D00 to U+1D7F </td></tr>
<tr>
<td><code>\p{IsPhoneticExtensionsSupplement}</code> </td><td>U+1D80 to U+1DBF </td></tr>
<tr>
<td><code>\p{IsCombiningDiacriticalMarksSupplement}</code> </td><td>U+1DC0 to U+1DFF </td></tr>
<tr>
<td><code>\p{IsLatinExtendedAdditional}</code> </td><td>U+1E00 to U+1EFF </td></tr>
<tr>
<td><code>\p{IsGreekExtended}</code> </td><td>U+1F00 to U+1FFF </td></tr>
<tr>
<td><code>\p{IsGeneralPunctuation}</code> </td><td>U+2000 to U+206F </td></tr>
<tr>
<td><code>\p{IsSuperscriptsandSubscripts}</code> </td><td>U+2070 to U+209F </td></tr>
<tr>
<td><code>\p{IsCurrencySymbols}</code> </td><td>U+20A0 to U+20CF </td></tr>
<tr>
<td><code>\p{IsCombiningDiacriticalMarksforSymbols}</code> </td><td>U+20D0 to U+20FF </td></tr>
<tr>
<td><code>\p{IsLetterlikeSymbols}</code> </td><td>U+2100 to U+214F </td></tr>
<tr>
<td><code>\p{IsNumberForms}</code> </td><td>U+2150 to U+218F </td></tr>
<tr>
<td><code>\p{IsArrows}</code> </td><td>U+2190 to U+21FF </td></tr>
<tr>
<td><code>\p{IsMathematicalOperators}</code> </td><td>U+2200 to U+22FF </td></tr>
<tr>
<td><code>\p{IsMiscellaneousTechnical}</code> </td><td>U+2300 to U+23FF </td></tr>
<tr>
<td><code>\p{IsControlPictures}</code> </td><td>U+2400 to U+243F </td></tr>
<tr>
<td><code>\p{IsOpticalCharacterRecognition}</code> </td><td>U+2440 to U+245F </td></tr>
<tr>
<td><code>\p{IsEnclosedAlphanumerics}</code> </td><td>U+2460 to U+24FF </td></tr>
<tr>
<td><code>\p{IsBoxDrawing}</code> </td><td>U+2500 to U+257F </td></tr>
<tr>
<td><code>\p{IsBlockElements}</code> </td><td>U+2580 to U+259F </td></tr>
<tr>
<td><code>\p{IsGeometricShapes}</code> </td><td>U+25A0 to U+25FF </td></tr>
<tr>
<td><code>\p{IsMiscellaneousSymbols}</code> </td><td>U+2600 to U+26FF </td></tr>
<tr>
<td><code>\p{IsDingbats}</code> </td><td>U+2700 to U+27BF </td></tr>
<tr>
<td><code>\p{IsMiscellaneousMathematicalSymbols-A}</code> </td><td>U+27C0 to U+27EF </td></tr>
<tr>
<td><code>\p{IsSupplementalArrows-A}</code> </td><td>U+27F0 to U+27FF </td></tr>
<tr>
<td><code>\p{IsBraillePatterns}</code> </td><td>U+2800 to U+28FF </td></tr>
<tr>
<td><code>\p{IsSupplementalArrows-B}</code> </td><td>U+2900 to U+297F </td></tr>
<tr>
<td><code>\p{IsMiscellaneousMathematicalSymbols-B}</code> </td><td>U+2980 to U+29FF </td></tr>
<tr>
<td><code>\p{IsSupplementalMathematicalOperators}</code> </td><td>U+2A00 to U+2AFF </td></tr>
<tr>
<td><code>\p{IsMiscellaneousSymbolsandArrows}</code> </td><td>U+2B00 to U+2BFF </td></tr>
<tr>
<td><code>\p{IsGlagolitic}</code> </td><td>U+2C00 to U+2C5F </td></tr>
<tr>
<td><code>\p{IsLatinExtended-C}</code> </td><td>U+2C60 to U+2C7F </td></tr>
<tr>
<td><code>\p{IsCoptic}</code> </td><td>U+2C80 to U+2CFF </td></tr>
<tr>
<td><code>\p{IsGeorgianSupplement}</code> </td><td>U+2D00 to U+2D2F </td></tr>
<tr>
<td><code>\p{IsTifinagh}</code> </td><td>U+2D30 to U+2D7F </td></tr>
<tr>
<td><code>\p{IsEthiopicExtended}</code> </td><td>U+2D80 to U+2DDF </td></tr>
<tr>
<td><code>\p{IsCyrillicExtended-A}</code> </td><td>U+2DE0 to U+2DFF </td></tr>
<tr>
<td><code>\p{IsSupplementalPunctuation}</code> </td><td>U+2E00 to U+2E7F </td></tr>
<tr>
<td><code>\p{IsCJKRadicalsSupplement}</code> </td><td>U+2E80 to U+2EFF </td></tr>
<tr>
<td><code>\p{IsKangxiRadicals}</code> </td><td>U+2F00 to U+2FDF </td></tr>
<tr>
<td><code>\p{IsIdeographicDescriptionCharacters}</code> </td><td>U+2FF0 to U+2FFF </td></tr>
<tr>
<td><code>\p{IsCJKSymbolsandPunctuation}</code> </td><td>U+3000 to U+303F </td></tr>
<tr>
<td><code>\p{IsHiragana}</code> </td><td>U+3040 to U+309F </td></tr>
<tr>
<td><code>\p{IsKatakana}</code> </td><td>U+30A0 to U+30FF </td></tr>
<tr>
<td><code>\p{IsBopomofo}</code> </td><td>U+3100 to U+312F </td></tr>
<tr>
<td><code>\p{IsHangulCompatibilityJamo}</code> </td><td>U+3130 to U+318F </td></tr>
<tr>
<td><code>\p{IsKanbun}</code> </td><td>U+3190 to U+319F </td></tr>
<tr>
<td><code>\p{IsBopomofoExtended}</code> </td><td>U+31A0 to U+31BF </td></tr>
<tr>
<td><code>\p{IsCJKStrokes}</code> </td><td>U+31C0 to U+31EF </td></tr>
<tr>
<td><code>\p{IsKatakanaPhoneticExtensions}</code> </td><td>U+31F0 to U+31FF </td></tr>
<tr>
<td><code>\p{IsEnclosedCJKLettersandMonths}</code> </td><td>U+3200 to U+32FF </td></tr>
<tr>
<td><code>\p{IsCJKCompatibility}</code> </td><td>U+3300 to U+33FF </td></tr>
<tr>
<td><code>\p{IsCJKUnifiedIdeographsExtensionA}</code> </td><td>U+3400 to U+4DBF </td></tr>
<tr>
<td><code>\p{IsYijingHexagramSymbols}</code> </td><td>U+4DC0 to U+4DFF </td></tr>
<tr>
<td><code>\p{IsCJKUnifiedIdeographs}</code> </td><td>U+4E00 to U+9FFF </td></tr>
<tr>
<td><code>\p{IsYiSyllables}</code> </td><td>U+A000 to U+A48F </td></tr>
<tr>
<td><code>\p{IsYiRadicals}</code> </td><td>U+A490 to U+A4CF </td></tr>
<tr>
<td><code>\p{IsLisu}</code> </td><td>U+A4D0 to U+A4FF </td></tr>
<tr>
<td><code>\p{IsVai}</code> </td><td>U+A500 to U+A63F </td></tr>
<tr>
<td><code>\p{IsCyrillicExtended-B}</code> </td><td>U+A640 to U+A69F </td></tr>
<tr>
<td><code>\p{IsBamum}</code> </td><td>U+A6A0 to U+A6FF </td></tr>
<tr>
<td><code>\p{IsModifierToneLetters}</code> </td><td>U+A700 to U+A71F </td></tr>
<tr>
<td><code>\p{IsLatinExtended-D}</code> </td><td>U+A720 to U+A7FF </td></tr>
<tr>
<td><code>\p{IsSylotiNagri}</code> </td><td>U+A800 to U+A82F </td></tr>
<tr>
<td><code>\p{IsCommonIndicNumberForms}</code> </td><td>U+A830 to U+A83F </td></tr>
<tr>
<td><code>\p{IsPhags-pa}</code> </td><td>U+A840 to U+A87F </td></tr>
<tr>
<td><code>\p{IsSaurashtra}</code> </td><td>U+A880 to U+A8DF </td></tr>
<tr>
<td><code>\p{IsDevanagariExtended}</code> </td><td>U+A8E0 to U+A8FF </td></tr>
<tr>
<td><code>\p{IsKayahLi}</code> </td><td>U+A900 to U+A92F </td></tr>
<tr>
<td><code>\p{IsRejang}</code> </td><td>U+A930 to U+A95F </td></tr>
<tr>
<td><code>\p{IsHangulJamoExtended-A}</code> </td><td>U+A960 to U+A97F </td></tr>
<tr>
<td><code>\p{IsJavanese}</code> </td><td>U+A980 to U+A9DF </td></tr>
<tr>
<td><code>\p{IsMyanmarExtended-B}</code> </td><td>U+A9E0 to U+A9FF </td></tr>
<tr>
<td><code>\p{IsCham}</code> </td><td>U+AA00 to U+AA5F </td></tr>
<tr>
<td><code>\p{IsMyanmarExtended-A}</code> </td><td>U+AA60 to U+AA7F </td></tr>
<tr>
<td><code>\p{IsTaiViet}</code> </td><td>U+AA80 to U+AADF </td></tr>
<tr>
<td><code>\p{IsMeeteiMayekExtensions}</code> </td><td>U+AAE0 to U+AAFF </td></tr>
<tr>
<td><code>\p{IsEthiopicExtended-A}</code> </td><td>U+AB00 to U+AB2F </td></tr>
<tr>
<td><code>\p{IsLatinExtended-E}</code> </td><td>U+AB30 to U+AB6F </td></tr>
<tr>
<td><code>\p{IsCherokeeSupplement}</code> </td><td>U+AB70 to U+ABBF </td></tr>
<tr>
<td><code>\p{IsMeeteiMayek}</code> </td><td>U+ABC0 to U+ABFF </td></tr>
<tr>
<td><code>\p{IsHangulSyllables}</code> </td><td>U+AC00 to U+D7AF </td></tr>
<tr>
<td><code>\p{IsHangulJamoExtended-B}</code> </td><td>U+D7B0 to U+D7FF </td></tr>
<tr>
<td><code>\p{IsHighSurrogates}</code> </td><td>U+D800 to U+DB7F </td></tr>
<tr>
<td><code>\p{IsHighPrivateUseSurrogates}</code> </td><td>U+DB80 to U+DBFF </td></tr>
<tr>
<td><code>\p{IsLowSurrogates}</code> </td><td>U+DC00 to U+DFFF </td></tr>
<tr>
<td><code>\p{IsPrivateUseArea}</code> </td><td>U+E000 to U+F8FF </td></tr>
<tr>
<td><code>\p{IsCJKCompatibilityIdeographs}</code> </td><td>U+F900 to U+FAFF </td></tr>
<tr>
<td><code>\p{IsAlphabeticPresentationForms}</code> </td><td>U+FB00 to U+FB4F </td></tr>
<tr>
<td><code>\p{IsArabicPresentationForms-A}</code> </td><td>U+FB50 to U+FDFF </td></tr>
<tr>
<td><code>\p{IsVariationSelectors}</code> </td><td>U+FE00 to U+FE0F </td></tr>
<tr>
<td><code>\p{IsVerticalForms}</code> </td><td>U+FE10 to U+FE1F </td></tr>
<tr>
<td><code>\p{IsCombiningHalfMarks}</code> </td><td>U+FE20 to U+FE2F </td></tr>
<tr>
<td><code>\p{IsCJKCompatibilityForms}</code> </td><td>U+FE30 to U+FE4F </td></tr>
<tr>
<td><code>\p{IsSmallFormVariants}</code> </td><td>U+FE50 to U+FE6F </td></tr>
<tr>
<td><code>\p{IsArabicPresentationForms-B}</code> </td><td>U+FE70 to U+FEFF </td></tr>
<tr>
<td><code>\p{IsHalfwidthandFullwidthForms}</code> </td><td>U+FF00 to U+FFEF </td></tr>
<tr>
<td><code>\p{IsSpecials}</code> </td><td>U+FFF0 to U+FFFF </td></tr>
</table>
<p>In addition, the <code>−−unicode</code> option enables standard Unicode language scripts:</p>
<p><code>\p{Adlam}</code>, <code>\p{Ahom}</code>, <code>\p{Anatolian_Hieroglyphs}</code>, <code>\p{Arabic}</code>, <code>\p{Armenian}</code>, <code>\p{Avestan}</code>, <code>\p{Balinese}</code>, <code>\p{Bamum}</code>, <code>\p{Bassa_Vah}</code>, <code>\p{Batak}</code>, <code>\p{Bengali}</code>, <code>\p{Bhaiksuki}</code>, <code>\p{Bopomofo}</code>, <code>\p{Brahmi}</code>, <code>\p{Braille}</code>, <code>\p{Buginese}</code>, <code>\p{Buhid}</code>, <code>\p{Canadian_Aboriginal}</code>, <code>\p{Carian}</code>, <code>\p{Caucasian_Albanian}</code>, <code>\p{Chakma}</code>, <code>\p{Cham}</code>, <code>\p{Cherokee}</code>, <code>\p{Chorasmian}</code>, <code>\p{Common}</code>, <code>\p{Coptic}</code>, <code>\p{Cuneiform}</code>, <code>\p{Cypriot}</code>, <code>\p{Cyrillic}</code>, <code>\p{Deseret}</code>, <code>\p{Devanagari}</code>, <code>\p{Dives_Akuru}</code>, <code>\p{Dogra}</code>, <code>\p{Duployan}</code>, <code>\p{Egyptian_Hieroglyphs}</code>, <code>\p{Elbasan}</code>, <code>\p{Elymaic}</code>, <code>\p{Ethiopic}</code>, <code>\p{Georgian}</code>, <code>\p{Glagolitic}</code>, <code>\p{Gothic}</code>, <code>\p{Grantha}</code>, <code>\p{Greek}</code>, <code>\p{Gujarati}</code>, <code>\p{Gunjala_Gondi}</code>, <code>\p{Gurmukhi}</code>, <code>\p{Han}</code>, <code>\p{Hangul}</code>, <code>\p{Hanifi_Rohingya}</code>, <code>\p{Hanunoo}</code>, <code>\p{Hatran}</code>, <code>\p{Hebrew}</code>, <code>\p{Hiragana}</code>, <code>\p{Imperial_Aramaic}</code>, <code>\p{Inscriptional_Pahlavi}</code>, <code>\p{Inscriptional_Parthian}</code>, <code>\p{Javanese}</code>, <code>\p{Kaithi}</code>, <code>\p{Kannada}</code>, <code>\p{Katakana}</code>, <code>\p{Kayah_Li}</code>, <code>\p{Kharoshthi}</code>, <code>\p{Khitan_Small_Script}</code>, <code>\p{Khmer}</code>, <code>\p{Khojki}</code>, <code>\p{Khudawadi}</code>, <code>\p{Lao}</code>, <code>\p{Latin}</code>, <code>\p{Lepcha}</code>, <code>\p{Limbu}</code>, <code>\p{Linear_A}</code>, <code>\p{Linear_B}</code>, <code>\p{Lisu}</code>, <code>\p{Lycian}</code>, <code>\p{Lydian}</code>, <code>\p{Mahajani}</code>, <code>\p{Makasar}</code>, <code>\p{Malayalam}</code>, <code>\p{Mandaic}</code>, <code>\p{Manichaean}</code>, <code>\p{Marchen}</code>, <code>\p{Masaram_Gondi}</code>, <code>\p{Medefaidrin}</code>, <code>\p{Meetei_Mayek}</code>, <code>\p{Mende_Kikakui}</code>, <code>\p{Meroitic_Cursive}</code>, <code>\p{Meroitic_Hieroglyphs}</code>, <code>\p{Miao}</code>, <code>\p{Modi}</code>, <code>\p{Mongolian}</code>, <code>\p{Mro}</code>, <code>\p{Multani}</code>, <code>\p{Myanmar}</code>, <code>\p{Nabataean}</code>, <code>\p{Nandinagari}</code>, <code>\p{New_Tai_Lue}</code>, <code>\p{Newa}</code>, <code>\p{Nko}</code>, <code>\p{Nushu}</code>, <code>\p{Nyiakeng_Puachue_Hmong}</code>, <code>\p{Ogham}</code>, <code>\p{Ol_Chiki}</code>, <code>\p{Old_Hungarian}</code>, <code>\p{Old_Italic}</code>, <code>\p{Old_North_Arabian}</code>, <code>\p{Old_Permic}</code>, <code>\p{Old_Persian}</code>, <code>\p{Old_Sogdian}</code>, <code>\p{Old_South_Arabian}</code>, <code>\p{Old_Turkic}</code>, <code>\p{Oriya}</code>, <code>\p{Osage}</code>, <code>\p{Osmanya}</code>, <code>\p{Pahawh_Hmong}</code>, <code>\p{Palmyrene}</code>, <code>\p{Pau_Cin_Hau}</code>, <code>\p{Phags_Pa}</code>, <code>\p{Phoenician}</code>, <code>\p{Psalter_Pahlavi}</code>, <code>\p{Rejang}</code>, <code>\p{Runic}</code>, <code>\p{Samaritan}</code>, <code>\p{Saurashtra}</code>, <code>\p{Sharada}</code>, <code>\p{Shavian}</code>, <code>\p{Siddham}</code>, <code>\p{SignWriting}</code>, <code>\p{Sinhala}</code>, <code>\p{Sogdian}</code>, <code>\p{Sora_Sompeng}</code>, <code>\p{Soyombo}</code>, <code>\p{Sundanese}</code>, <code>\p{Syloti_Nagri}</code>, <code>\p{Syriac}</code>, <code>\p{Tagalog}</code>, <code>\p{Tagbanwa}</code>, <code>\p{Tai_Le}</code>, <code>\p{Tai_Tham}</code>, <code>\p{Tai_Viet}</code>, <code>\p{Takri}</code>, <code>\p{Tamil}</code>, <code>\p{Tangut}</code>, <code>\p{Telugu}</code>, <code>\p{Thaana}</code>, <code>\p{Thai}</code>, <code>\p{Tibetan}</code>, <code>\p{Tifinagh}</code>, <code>\p{Tirhuta}</code>, <code>\p{Ugaritic}</code>, <code>\p{Vai}</code>, <code>\p{Wancho}</code>, <code>\p{Warang_Citi}</code>, <code>\p{Yezidi}</code>, <code>\p{Yi}</code>, <code>\p{Zanabazar_Square}</code>,</p>
<dl class="section note"><dt>Note</dt><dd>Unicode language script character classes differ from the Unicode blocks that have a similar name. For example, the <code>\p{Greek}</code> class represents Greek and Coptic letters and differs from the Unicode block <code>\p{IsGreek}</code> that spans a specific Unicode block of Greek and Coptic characters only, which also includes unassigned characters.</dd></dl>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-anchor"></a>
Anchors and boundaries</h3>
<p>Anchors are used to demarcate the start and end of input or the start and end of a line:</p>
<table class="doxtable">
<tr>
<th>Pattern </th><th>Matches  </th></tr>
<tr>
<td><code>^φ</code> </td><td>matches <code>φ</code> at the start of input or start of a line (multi-line mode) </td></tr>
<tr>
<td><code>φ$</code> </td><td>matches <code>φ</code> at the end of input or end of a line (multi-line mode) </td></tr>
<tr>
<td><code>\Aφ</code> </td><td>matches <code>φ</code> at the start of input </td></tr>
<tr>
<td><code>φ\z</code> </td><td>matches <code>φ</code> at the end of input </td></tr>
</table>
<p>Anchors in lexer specifications require pattern context, meaning that <code>φ</code> cannot be empty.</p>
<p>Note that <code>&lt;&lt;EOF&gt;&gt;</code> in lexer specifications match the end of input, which can be used in place of the pattern <code>\z</code>.</p>
<p>Actions for the start of input can be specified in an initial code block preceding the rules, see <a class="el" href="index.html#reflex-code-blocks">Initial code blocks </a>.</p>
<p>Word boundaries demarcate words. Word characters <code>\w</code> are letters, digits, and the underscore.</p>
<table class="doxtable">
<tr>
<th>Pattern </th><th>Matches  </th></tr>
<tr>
<td><code>\bφ</code> </td><td>matches <code>φ</code> starting at a word boundary </td></tr>
<tr>
<td><code>φ\b</code> </td><td>matches <code>φ</code> ending at a word boundary </td></tr>
<tr>
<td><code>\Bφ</code> </td><td>matches <code>φ</code> starting at a non-word boundary </td></tr>
<tr>
<td><code>φ\B</code> </td><td>matches <code>φ</code> ending at a non-word boundary </td></tr>
<tr>
<td><code>\&lt;φ</code> </td><td>matches <code>φ</code> that starts as a word </td></tr>
<tr>
<td><code>\&gt;φ</code> </td><td>matches <code>φ</code> that starts as a non-word </td></tr>
<tr>
<td><code>φ\&lt;</code> </td><td>matches <code>φ</code> that ends as a non-word </td></tr>
<tr>
<td><code>φ\&gt;</code> </td><td>matches <code>φ</code> that ends as a word </td></tr>
</table>
<dl class="section note"><dt>Note</dt><dd>The RE/flex regex library requires anchors and word boundaries to be specified in patterns at the start or end of the pattern. Boundaries are not permitted in the middle of a pattern, see <a class="el" href="index.html#reflex-limitations">Limitations </a>.</dd></dl>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-dents"></a>
Indent/nodent/dedent</h3>
<p>Automatic indent and dedent matching is a special feature of RE/flex and is only available when the RE/flex matcher engine is used (the default matcher). An indent and a dedent position is defined and matched with:</p>
<table class="doxtable">
<tr>
<th>Pattern </th><th>Matches  </th></tr>
<tr>
<td><code>\i</code> </td><td>indent: matches and adds a new indent stop position </td></tr>
<tr>
<td><code>\j</code> </td><td>dedent: matches a previous indent position, removes one indent stop </td></tr>
</table>
<p>The <code>\i</code> and <code>\j</code> anchors should be used in combination with the start of a line anchor <code>^</code> followed by a pattern that represents left margin spacing for indentations, followed by a <code>\i</code> or a <code>\j</code> at the end of the pattern. The margin spacing pattern may include any characters that are considered part of the left margin, but should exclude <code>\n</code>. For example:</p>
<div class="alt"> <div class="fragment"><div class="line">%o tabs=8</div><div class="line">%%</div><div class="line">^\h+      out() &lt;&lt; <span class="stringliteral">&quot;| &quot;</span>; <span class="comment">// nodent: text is aligned to current indent</span></div><div class="line">^\h+\i    out() &lt;&lt; <span class="stringliteral">&quot;&gt; &quot;</span>; <span class="comment">// indent: matched and added with \i</span></div><div class="line">^\h*\j    out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>; <span class="comment">// dedent: matched with \j</span></div><div class="line">\j        out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>; <span class="comment">// dedent: for each extra level dedented</span></div><div class="line">.|\n      echo();</div><div class="line">%%</div></div><!-- fragment --> </div><p>The <code>\h</code> pattern matches space and tabs, where tabs advance to the next column that is a multiple of 8. The tab multiplier can be changed by setting the <code>−−tabs=N</code> option where <code>N</code> must be 1, 2, 4, or 8. The tabs value can be changed at runtime with <code>matcher().tabs(N)</code>:</p>
<table class="doxtable">
<tr>
<th>RE/flex action </th><th>Result  </th></tr>
<tr>
<td><code>matcher().tabs()</code> </td><td>returns the current tabs value 1, 2, 4, or 8 </td></tr>
<tr>
<td><code>matcher().tabs(n)</code> </td><td>set the tabs value <code>n</code> where <code>n</code> is 1, 2, 4 or 8 </td></tr>
</table>
<p>Using negative patterns we can ignore empty lines and multi-line comments that would otherwise affect indent stops:</p>
<div class="alt"> <div class="fragment"><div class="line">%o <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a> tabs=8</div><div class="line">%%</div><div class="line">^\h+                      out() &lt;&lt; <span class="stringliteral">&quot;| &quot;</span>; <span class="comment">// nodent, text is aligned to current margin column</span></div><div class="line">^\h+\i                    out() &lt;&lt; <span class="stringliteral">&quot;&gt; &quot;</span>; <span class="comment">// indent</span></div><div class="line">^\h*\j                    out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>; <span class="comment">// dedent</span></div><div class="line">\j                        out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>; <span class="comment">// dedent, for each extra level dedented</span></div><div class="line">(?^^\h*\n)                <span class="comment">// eat empty lines without affecting indent stops</span></div><div class="line">(?^^\h*<span class="stringliteral">&quot;/*&quot;</span>(.|\n)*?<span class="stringliteral">&quot;*/&quot;</span>)  <span class="comment">// eat /*-comments that start a line without affecting indent stops</span></div><div class="line">(?^\\\n\h*)               <span class="comment">// lines ending in \ continue on the next line</span></div><div class="line">(?^<span class="stringliteral">&quot;/*&quot;</span>(.|\n)*?<span class="stringliteral">&quot;*/&quot;</span>)      <span class="comment">// eat /*-comments</span></div><div class="line">.|\n                      echo(); <span class="comment">// ECHO character</span></div><div class="line">%%</div></div><!-- fragment --> </div><p>Likewise, we can add rules to ignore inline <code>//</code>-comments to our lexer specification. To do so, we should add a rule with pattern <code>(?^^\h*"//".*)</code> to ignore <code>//</code>-comments without affecting stop positions.</p>
<p>To scan input that continues on the next new line(s) (which may affect indent stops) while preserving the current indent stop positions, use the RE/flex matcher <code>matcher().push_stops()</code> and <code>matcher().pop_stops()</code>, or <code>matcher().stops()</code> to directlye access the vector of indent stops to modify:</p>
<table class="doxtable">
<tr>
<th>RE/flex action </th><th>Result  </th></tr>
<tr>
<td><code>matcher().push_stops()</code> </td><td>push indent stops on the stack then clear stops </td></tr>
<tr>
<td><code>matcher().pop_stops()</code> </td><td>pop indent stops and make them current </td></tr>
<tr>
<td><code>matcher().clear_stops()</code> </td><td>clear current indent stops </td></tr>
<tr>
<td><code>matcher().stops()</code> </td><td>reference to current <code>std::vector&lt;size_t&gt;</code> stops </td></tr>
<tr>
<td><code>matcher().last_stop()</code> </td><td>returns the last indent stop position or 0 </td></tr>
<tr>
<td><code>matcher().insert_stop(n)</code> </td><td>inserts/appends an indent stop at position <code>n</code> </td></tr>
<tr>
<td><code>matcher().delete_stop(n)</code> </td><td>remove stop positions from position <code>n</code> and up </td></tr>
</table>
<p>For example, to continue scanning after a <code>/*</code> for multiple lines without indentation matching, allowing for possible nested <code>/*</code>-comments, up to a <code>*/</code> you can save the current indent stop positions and transition to a new start condition state to scan the content between <code>/*</code> and <code>*/</code>:</p>
<div class="alt"> <div class="fragment"><div class="line">%{</div><div class="line">  <span class="keywordtype">int</span> level;                 <span class="comment">// a variable to track the /*-comment nesting level</span></div><div class="line">  std::vector&lt;size_t&gt; stops; <span class="comment">// a variable to save the stop positions after indent</span></div><div class="line">%}</div><div class="line">%o tabs=8</div><div class="line">%x COMMENT</div><div class="line">%%</div><div class="line">^\h+           out() &lt;&lt; <span class="stringliteral">&quot;| &quot;</span>;               <span class="comment">// nodent, text is aligned to current margin column</span></div><div class="line">^\h+\i         out() &lt;&lt; <span class="stringliteral">&quot;&gt; &quot;</span>;               <span class="comment">// indent</span></div><div class="line">               stops = matcher().stops();   <span class="comment">// save the stop positions</span></div><div class="line">^<span class="stringliteral">&quot;/*&quot;</span>\j        level = 1;                   <span class="comment">// do not count dedent(s) to the first line that has a /*-comment</span></div><div class="line">               start(COMMENT);              <span class="comment">// skip comment</span></div><div class="line">^\h*\j         out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>;               <span class="comment">// dedent</span></div><div class="line">\j             out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>;               <span class="comment">// dedent, triggered for each extra level dedented</span></div><div class="line">(?^^\h*\n)     <span class="comment">// a negative pattern to eat empty lines without affecting indent stops</span></div><div class="line">(?^^\h+/<span class="stringliteral">&quot;/*&quot;</span>)  <span class="comment">// a negative pattern to eat white space before /*-comments without affecting indent stops</span></div><div class="line"><span class="stringliteral">&quot;/*&quot;</span>           level = 1;</div><div class="line">               start(COMMENT);              <span class="comment">// continue w/o indent matching</span></div><div class="line">(?^\\\n\h*)    <span class="comment">// lines ending in \ continue on the next line</span></div><div class="line">.|\n           echo();                      <span class="comment">// ECHO character</span></div><div class="line">&lt;COMMENT&gt;{</div><div class="line"><span class="stringliteral">&quot;/*&quot;</span>           ++level;                     <span class="comment">// allow nested /*-comments </span></div><div class="line"><span class="stringliteral">&quot;*/&quot;</span>           <span class="keywordflow">if</span> (--level == 0)</div><div class="line">               {</div><div class="line">                 matcher().stops() = stops; <span class="comment">// restore the indent margin/tab stops</span></div><div class="line">                 start(INITIAL);            <span class="comment">// back to initial state</span></div><div class="line">               }</div><div class="line">.|\n           <span class="comment">// ignore all content in comments</span></div><div class="line">&lt;&lt;EOF&gt;&gt;        out() &lt;&lt; <span class="stringliteral">&quot;/* not closed&quot;</span>;</div><div class="line">}</div><div class="line">%%</div></div><!-- fragment --> </div><p>The multi-line comments enclosed in <code>/*</code> <code>*/</code> are processed by the exclusive <code>COMMENT</code> start condition rules. The rules allow for <code>/*</code>-comment nesting. We use <code>stops = matcher().stops()</code> and <code>matcher().stops() = stops</code> to save and restore stops.</p>
<p>In this example we added rules so that comments on a line do not affect the current indent stops. This is done by using the negative pattern <code>(?^^\h+/"/*")</code> with a trailing context <code>/"/*"</code>. Here we used a negative pattern to eat the margin spacing without affecting indent stops. The trailing context looks ahead for a <code>/*</code> but does not consume the <code>/*</code>.</p>
<p>However, when a <code>/*</code>-comment starts at the first column of a line, the pattern <code>(?^^\h+/"/*")</code> does not match it, even when we change it to <code>(?^^\h*/"/*")</code>. This is because the <code>\h*</code> cannot be an empty match since the trailing context does not return a match, and matches cannot be empty. Therefore, adding the rule with pattern <code>^"/*"\j</code> adjusts for that, but accepting the dedents caused by the <code>/*</code>-comment. This is fine, because the stop positions are restored after scanning the <code>/*</code>-comment.</p>
<p>We added the negative pattern <code>(?^^\h*\n)</code> to ignore empty lines. This allows empty lines in the input without affecting indent stops.</p>
<dl class="section warning"><dt>Warning</dt><dd>When using the <code>matcher().stops()</code> method to access the vector of stops to modify, we must make sure to keep the stop positions in the vector sorted.</dd></dl>
<p>In addition to the <code>\i</code> and <code>\j</code> indent and dedent anchors, the <code>\k</code> undent anchor matches when the indent depth changed (before the position of <code>\k</code>), undoing this change to keep the current indent stops ("undenting"):</p>
<table class="doxtable">
<tr>
<th>Pattern </th><th>Matches  </th></tr>
<tr>
<td><code>\k</code> </td><td>undent: matches when indent depth changed, keep current indent stops </td></tr>
</table>
<p>The example shown above can be simplified with <code>\k</code>. We no longer need to explicitly save and restore indent stops in a variable:</p>
<div class="alt"> <div class="fragment"><div class="line">%{</div><div class="line">  <span class="keywordtype">int</span> level;   <span class="comment">// a variable to track the /*-comment nesting level</span></div><div class="line">%}</div><div class="line">%o tabs=8</div><div class="line">%x COMMENT</div><div class="line">%%</div><div class="line">^\h+           out() &lt;&lt; <span class="stringliteral">&quot;| &quot;</span>;    <span class="comment">// nodent, text is aligned to current margin column</span></div><div class="line">^\h+\i         out() &lt;&lt; <span class="stringliteral">&quot;&gt; &quot;</span>;    <span class="comment">// indent</span></div><div class="line">^\h*\j         out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>;    <span class="comment">// dedent</span></div><div class="line">\j             out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>;    <span class="comment">// dedent, triggered for each extra level dedented</span></div><div class="line">(?^^\h*\n)     <span class="comment">// a negative pattern to eat empty lines without affecting indent stops</span></div><div class="line">\h*<span class="stringliteral">&quot;/*&quot;</span>\k?     level = 1;        <span class="comment">// /*-comment after spacing, \k matches indent stop changes</span></div><div class="line">               start(COMMENT);   <span class="comment">// continue w/o indent matching</span></div><div class="line">(?^\\\n\h*)    <span class="comment">// lines ending in \ continue on the next line</span></div><div class="line">.|\n           echo();           <span class="comment">// ECHO character</span></div><div class="line">&lt;COMMENT&gt;{</div><div class="line"><span class="stringliteral">&quot;/*&quot;</span>           ++level;          <span class="comment">// allow nested /*-comments </span></div><div class="line"><span class="stringliteral">&quot;*/&quot;</span>           <span class="keywordflow">if</span> (--level == 0)</div><div class="line">                 start(INITIAL); <span class="comment">// back to initial state</span></div><div class="line">.|\n           <span class="comment">// ignore all content in comments</span></div><div class="line">&lt;&lt;EOF&gt;&gt;        out() &lt;&lt; <span class="stringliteral">&quot;/* not closed&quot;</span>;</div><div class="line">}</div><div class="line">%%</div></div><!-- fragment --> </div><p>The pattern <code>\h*"/*"\k?</code> matches a <code>/*</code>-comment with leading white space. The <code>\k</code> anchor matches if the indent depth changed in the leading white space, which is also matched by the first three patterns in the lexer specification before their <code>\i</code> and <code>\j</code> indent and dedent anchors, respectively. If the indent depth changed, the <code>\k</code> anchor matches, while keeping the current indent stops unchanged by undoing these changes. Because we also want to match <code>\*</code> when the indent depth does not change, we made <code>\k</code> optional in pattern <code>\h*"/*"\k?</code>. The anchor <code>^</code> is not used here either, since comments after any spacing should be matched. Alternatively, two patterns <code>^\h*"/*"\k</code> and <code>\h*"/*"</code> may be used, where the first matches if and only if the indent stops changed on a new line and were undone.</p>
<p>Note that the <code>COMMENT</code> rules do not use <code>\i</code> or <code>\j</code>. This means that the current indent stops are never matched or changed and remain the same as in the <code>INITIAL</code> state, when returning to the <code>INITIAL</code> state.</p>
<p>Another use of <code>\k</code> is to ignore indents to only detect a closing dedent with <code>\j</code>. For example, when comments are allowed to span multiple lines when indented below the start of the <code>#</code> comment:</p>
<div class="alt"> <div class="fragment"><div class="line">%x COMMENT MORECOM</div><div class="line">%%</div><div class="line">^\h+\i         <span class="comment">// indent</span></div><div class="line">^\h*\j         <span class="comment">// dedent</span></div><div class="line">\j             <span class="comment">// dedent</span></div><div class="line">^\h+           <span class="comment">// nodent</span></div><div class="line"><span class="preprocessor">#.*\n          start(COMMENT);</span></div><div class="line">&lt;COMMENT&gt;{</div><div class="line">^\h+\i         start(MORECOM);</div><div class="line">.|\n           matcher().less(0); start(INITIAL);</div><div class="line">}</div><div class="line">&lt;MORECOM&gt;{</div><div class="line">^\h*\j         |</div><div class="line">\j             start(INITIAL);</div><div class="line">^\h+\k         <span class="comment">// undent, i.e. ignore all indents in comments</span></div><div class="line">.|\n           <span class="comment">// ignore all content in comments</span></div><div class="line">}</div><div class="line">.|\n           echo();</div><div class="line">%%</div></div><!-- fragment --> </div><p>The <code>COMMENT</code> state checks for an indent to switch to state <code>MORECOM</code>, which eats the indented comment block. When there is no indent <code>.|\n</code> is matched, i.e. something must be matched. This match is put back into the input with <code>matcher().less(0)</code> (or <code><a class="el" href="flexlexer_8h.html#a6e9a1ca8eb82e1d2dea4fa4a4dfc5c07" title="Flex-compatible macro: truncate the yytext length of the match to n characters in length and repositi...">yyless(0)</a></code> with <code>−−flex</code>).</p>
<p>Alternatively, the indent level in the <code>COMMENT</code> rules could be tracked by incrementing a variable when matching <code>\i</code> and decrementing the variable when matching <code>\j</code> until the variable is zero at the final dedent.</p>
<dl class="section note"><dt>Note</dt><dd>Anchors <code>\i</code>, <code>\j</code>, and <code>\k</code> should appear at the end of a regex pattern.</dd></dl>
<p>See <a class="el" href="index.html#reflex-states">Start condition states </a> for more information about start condition states. See <a class="el" href="index.html#reflex-pattern-negative">Negative patterns</a> for more information on negative patterns.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-negative"></a>
Negative patterns</h3>
<p>When negative patterns of the form <code>(?^φ)</code> match, they are simply ignored by the matcher and never returned as matches. They are useful to return matches for some given pattern except when this pattern is more specific. For example, to match any sequence of digits except digits starting with a zero the pattern <code>\d+|(?^0\d+)</code> can be used instead of <code>[1-9]\d+</code>. While these two patterns may look similar at first glance, these two patterns differ in that the first pattern (with the negative sub-pattern <code>(?^0\d+)</code>) ignores numbers with leading zeros such as <code>012</code> while the second pattern will match the <code>12</code> in <code>012</code>.</p>
<p>As another example, say we are searching for a given word while ignoring occurrences of the word in quoted strings. We can use the pattern <code>word|(?^".*?")</code> for this, where <code>(?^".*?")</code> matches all quoted strings that we want to ignore (to skip C/C++ quoted strings in source code input files, use the longer pattern <code>(?^"(\\\\.|\\\\\\r?\\n|[^\\\\\\n"])*")</code>).</p>
<p>A negative pattern can also be used to consume line continuations without affecting the indentation stops defined by indent marker <code>\i</code>. Negative patterns are a RE/flex feature. For example:</p>
<div class="alt"> <div class="fragment"><div class="line">%o tabs=8</div><div class="line">%%</div><div class="line">^\h+         out() &lt;&lt; <span class="stringliteral">&quot;| &quot;</span>; <span class="comment">// nodent: text is aligned to current indent</span></div><div class="line">^\h*\i       out() &lt;&lt; <span class="stringliteral">&quot;&gt; &quot;</span>; <span class="comment">// indent: matched and added with \i</span></div><div class="line">^\h*\j       out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>; <span class="comment">// dedent: matched with \j</span></div><div class="line">\j           out() &lt;&lt; <span class="stringliteral">&quot;&lt; &quot;</span>; <span class="comment">// dedent: for each extra level dedented</span></div><div class="line">(?^\\\n\h+)  <span class="comment">/* lines ending in \ will continue on the next line</span></div><div class="line"><span class="comment">                without affecting the current \i stop positions */</span></div><div class="line">.|\n         echo();</div><div class="line">%%</div></div><!-- fragment --> </div><p>The negative pattern <code>(?^\\\n\h+)</code> consumes input internally as if we are repeately calling <code>input()</code> (or <code><a class="el" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a" title="Flex-compatible macro: read one 8-bit character, returns zero when EOF. ">yyinput()</a></code> with <code>−−flex</code>). We used it here to consume the line-ending <code>\</code> and the indent that followed it, as if this text was not part of the input, which ensures that the current indent positions defined by <code>\i</code> are not affected. See <a class="el" href="index.html#reflex-pattern-dents">Indent/nodent/dedent</a> for more details on indentation matching.</p>
<p>Note that any actions corresponding to negative patterns in the lexer specification are never executed, because negative pattern matches are never returned by the matcher engine.</p>
<dl class="section warning"><dt>Warning</dt><dd>Negative patterns may be preceded by any pattern to enlarge the negative pattern. That is, <code>X(?^Y)</code> equals <code>(?^XY)</code>. However, when followed by a pattern <code>(?^X)Y</code> the matching behavior is currently not well defined. For example <code>(?^ab)c</code> matches <code>abc</code> but ignores input <code>ab</code> when not followed by a <code>c</code>, as if the pattern was <code>(?^ab)c?</code>. Future RE/flex updates will meet the requirement that patterns of the form <code>(?^X)Y</code> equal <code>(?^XY)</code>.</dd></dl>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-lookahead"></a>
Lookahead</h3>
<p>A lookahead pattern <code>φ(?=ψ)</code> matches <code>φ</code> only when followed by pattern <code>ψ</code>. The text matched by <code>ψ</code> is not consumed.</p>
<p>Boost.Regex and PCRE2 matchers support lookahead <code>φ(?=ψ)</code> and lookbehind <code>φ(?&lt;=ψ)</code> patterns that may appear anywhere in a regex. The RE/flex matcher supports lookahead at the end of a pattern, similar to <a class="el" href="index.html#reflex-pattern-trailing">Trailing context</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-lookbehind"></a>
Lookbehind</h3>
<p>A lookbehind pattern <code>φ(?&lt;=ψ)</code> matches <code>φ</code> only when it also matches pattern <code>ψ</code> at its end (that is, <code>.*(?&lt;=ab)</code> matches anything that ends in <code>ab</code>).</p>
<p>The RE/flex matcher does not support lookbehind. Lookbehind patterns should not look too far behind, see <a class="el" href="index.html#reflex-limitations">Limitations </a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-trailing"></a>
Trailing context</h3>
<p>Flex "trailing context" <code>φ/ψ</code> matches a pattern <code>φ</code> only when followed by the lookahead pattern <code>φ</code>. A trailing context <code>φ/ψ</code> has the same meaning as the lookahead <code>φ(?=ψ)</code>, see <a class="el" href="index.html#reflex-pattern-lookahead">Lookahead</a>.</p>
<p>A trailing context can only be used in lexer specifications and should only occur at the end of a pattern, not in the middle of a pattern. There are some important <a class="el" href="index.html#reflex-limitations">Limitations </a> to consider that are historical and related to the construction of efficient FSMs for regular expressions. The limitations apply to trailing context and lookaheads that the RE/flex matcher implements.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-unicode"></a>
Unicode mode</h3>
<p>Use <b><code>reflex</code></b> option <code>−−unicode</code> (or <em><code>%option unicode</code></em>) to globally enable Unicode. Use <code>(?u:φ)</code> to locally enable Unicode in a pattern <code>φ</code>. Use <code>(?-u:φ)</code> to locally disable Unicode in <code>φ</code>. Unicode mode enables the following patterns to be used:</p>
<table class="doxtable">
<tr>
<th>Pattern </th><th>Matches  </th></tr>
<tr>
<td><code>.</code> </td><td>matches any Unicode character (beware of <a class="el" href="index.html#invalid-utf">Invalid UTF encodings </a>) </td></tr>
<tr>
<td><code>€</code> (UTF-8) </td><td>matches wide character <code>€</code>, encoded in UTF-8 </td></tr>
<tr>
<td><code>[€¥£]</code> (UTF-8) </td><td>matches wide character <code>€</code>, <code>¥</code> or <code>£</code>, encoded in UTF-8 </td></tr>
<tr>
<td><code>\X</code> </td><td>matches any ISO-8859-1 or Unicode character </td></tr>
<tr>
<td><code>\R</code> </td><td>matches a Unicode line break <code>\r\n</code> or <code>[\u{000A}-\u{000D}u{U+0085}\u{2028}\u{2029}]</code> </td></tr>
<tr>
<td><code>\s</code> </td><td>matches a white space character with Unicode sub-property Zs </td></tr>
<tr>
<td><code>\l</code> </td><td>matches a lower case letter with Unicode sub-property Ll </td></tr>
<tr>
<td><code>\u</code> </td><td>matches an upper case letter with Unicode sub-property Lu </td></tr>
<tr>
<td><code>\w</code> </td><td>matches a Unicode word character with property L, Nd, or Pc </td></tr>
<tr>
<td><code>\u{20AC}</code> </td><td>matches Unicode character U+20AC </td></tr>
<tr>
<td><code>\p{C}</code> </td><td>matches a character in category C </td></tr>
<tr>
<td><code>\p{^C}</code>,<code>\P{C}</code> </td><td>matches any character except in category C </td></tr>
</table>
<p>When converting regex patterns for use with a C++ regex library, use regex matcher converter flag <code><a class="el" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39" title="convert . (dot), \s, \w, \l, \u, \S, \W, \L, \U to Unicode ">reflex::convert_flag::unicode</a></code> to convert Unicode patterns for use with the 8-bit based RE/flex, Boost.Regex, PCRE2, and std::regex regex libraries, see <a class="el" href="index.html#regex-convert">Regex converters </a> for more details.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-freespace"></a>
Free space mode</h3>
<p>Free space mode can be useful to improve readability of patterns. Free space mode permits spacing between concatenations and alternations in patterns. To to match a single space use <code>[ ]</code>, to match a tab use <code>[\t]</code>, to match either use <code>\h</code>. Long patterns may continue on the next line when the line ends with an escape <code>\</code>. Comments are ignored in patterns in free-space mode. Comments start with a <code>#</code> and end at the end of the line. To specify a <code>#</code> use <code>[#]</code>.</p>
<p>In addition, <code>/*...*/</code> comments are permitted in lexer specifications in free-space mode when the <code>−−matcher=reflex</code> option is specified (the default matcher).</p>
<p>Free space mode requires lexer actions in <a class="el" href="index.html#reflex-spec-rules">The rules section</a> of a lexer specification to be placed in <em><code>{ }</code></em> blocks and user code to be placed in <em><code>%{ %}</code></em> blocks instead of indented.</p>
<p>To enable free space mode in <b><code>reflex</code></b> use the <code>−−freespace</code> option (or <em><code>%option freespace</code></em>).</p>
<p>Prepend <code>(?x)</code> to the regex to specify free-space mode or use <code>(?x:φ)</code> to locally enable free-space mode in the sub-pattern <code>φ</code>. Use <code>(?-x:φ)</code> to locally disable free-space mode in <code>φ</code>. The regex pattern may require conversion when the regex library does not support free-space mode modifiers, see <a class="el" href="index.html#regex-convert">Regex converters </a> for more details.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-multiline"></a>
Multi-line mode</h3>
<p>Multi-line mode makes the anchors <code>^</code> and <code>$</code> match the start and end of a line, respectively. Multi-line mode is the default mode in lexer specifications.</p>
<p>Prepend <code>(?m)</code> to the regex to specify multi-line mode or use <code>(?m:φ)</code> to locally enable multi-line mode in the sub-pattern <code>φ</code>. Use <code>(?-m:φ)</code> to locally disable multi-line mode in <code>φ</code>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-dotall"></a>
Dotall mode</h3>
<p>To enable dotall mode in <b><code>reflex</code></b> use the <code>-a</code> or <code>−−dotall</code> option (or <em><code>%option dotall</code></em>).</p>
<p>Prepend <code>(?s)</code> to the regex to specify dotall mode or use <code>(?s:φ)</code> to locally enable dotall mode in the sub-pattern <code>φ</code>. Use <code>(?-s:φ)</code> to locally disable dotall mode in <code>φ</code>. The regex pattern may require conversion when the regex library does not support dotall mode modifiers, see <a class="el" href="index.html#regex-convert">Regex converters </a> for more details.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-anycase"></a>
Case-insensitive mode</h3>
<p>To enable case-insensitive mode in <b><code>reflex</code></b> use the <code>-i</code> or <code>−−case-insensitive</code> option (or <em><code>%option case-insensitive</code></em>).</p>
<p>Prepend <code>(?i)</code> to the regex to specify case-insensitive mode or use <code>(?i:φ)</code> to locally enable case-insensitive mode in the sub-pattern <code>φ</code>. Use <code>(?-i:φ)</code> to locally disable case-insensitive mode in <code>φ</code>. The regex pattern may require conversion when the regex library does not support case-insensitive mode modifiers, see <a class="el" href="index.html#regex-convert">Regex converters </a> for more details.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-pattern-modifiers"></a>
Multiple mode modifiers</h3>
<p>Multiple <code>(?i:φ)</code> <a class="el" href="index.html#reflex-pattern-anycase">Case-insensitive mode</a>, <code>(?m:φ)</code> <a class="el" href="index.html#reflex-pattern-multiline">Multi-line mode</a>, <code>(?s:φ)</code> <a class="el" href="index.html#reflex-pattern-dotall">Dotall mode</a>, <code>(?u:φ)</code> <a class="el" href="index.html#reflex-pattern-unicode">Unicode mode</a>, and <code>(?x:φ)</code> <a class="el" href="index.html#reflex-pattern-freespace">Free space mode</a> modifiers may be applied to the same pattern <code>φ</code> by combining them in one inline modifier <code>(?imsux-imsux:φ)</code>, where the mode modifiers before the dash are enabled and the mode modifiers after the dash are disabled.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-lexer"></a>
The Lexer/yyFlexLexer class                                     </h2>
<p>By default, <b><code>reflex</code></b> produces a Lexer class with a virtual lex scanner function. The name of this function as well as the Lexer class name and the namespace can be set with options:</p>
<table class="doxtable">
<tr>
<th>Option </th><th>RE/flex default name </th><th>Flex default name  </th></tr>
<tr>
<td><code>namespace</code> </td><td><em>n/a</em> </td><td><em>n/a</em> </td></tr>
<tr>
<td><code>lexer</code> </td><td><code>Lexer</code> class </td><td><code>yyFlexLexer</code> class </td></tr>
<tr>
<td><code>lex</code> </td><td><code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> function </td><td><code>yylex()</code> function </td></tr>
</table>
<p>To customize the Lexer class use these options and code injection.</p>
<p>You can declare multiple nested namespace names by <code>namespace=NAME1::NAME2::NAME3</code>, or by separating the names with a dot such as <code>namespace=NAME1.NAME2.NAME3</code>, to declare the lexer in <code>NAME1::NAME2::NAME3</code>.</p>
<p>To understand the impact of these options, consider the following lex specification template:</p>
<div class="alt"> <div class="fragment"><div class="line">%option <span class="keyword">namespace</span>=NAMESPACE</div><div class="line">%option lexer=LEXER</div><div class="line">%option <a class="code" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805">lex</a>=LEX</div><div class="line"></div><div class="line">%<span class="keyword">class</span>{</div><div class="line">  MEMBERS</div><div class="line">}</div><div class="line"></div><div class="line">%option ctorarg=<span class="stringliteral">&quot;CTORARG&quot;</span></div><div class="line">%init{</div><div class="line">  INIT</div><div class="line">}</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">%{</div><div class="line">  CODE</div><div class="line">%}</div><div class="line"></div><div class="line">REGEX ACTION</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>This produces the following Lexer class with the template parts filled in:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="abslexer_8h.html">reflex/abslexer.h</a>&gt;</span></div><div class="line"><span class="keyword">namespace </span>NAMESPACE {</div><div class="line">  <span class="keyword">class </span>LEXER : <span class="keyword">public</span> <a class="code" href="classreflex_1_1_abstract_lexer.html">reflex::AbstractLexer</a>&lt;reflex::Matcher&gt; {</div><div class="line">    MEMBERS</div><div class="line">   <span class="keyword">public</span>:</div><div class="line">    LEXER(</div><div class="line">        CTORARG,</div><div class="line">        <span class="keyword">const</span> <a class="code" href="classreflex_1_1_input.html">reflex::Input</a>&amp; input = <a class="code" href="classreflex_1_1_input.html">reflex::Input</a>(),</div><div class="line">        std::ostream&amp;        os    = std::cout)</div><div class="line">      :</div><div class="line">        AbstractLexer(input, os)</div><div class="line">    {</div><div class="line">      INIT</div><div class="line">    }</div><div class="line">    <span class="keyword">static</span> <span class="keyword">const</span> <span class="keywordtype">int</span> INITIAL = 0;</div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">int</span> LEX(<span class="keywordtype">void</span>);</div><div class="line">    <span class="keywordtype">int</span> LEX(</div><div class="line">        <span class="keyword">const</span> <a class="code" href="classreflex_1_1_input.html">reflex::Input</a>&amp; input,</div><div class="line">        std::ostream        *os = NULL)</div><div class="line">    {</div><div class="line">      in(input);</div><div class="line">      <span class="keywordflow">if</span> (os)</div><div class="line">        out(*os);</div><div class="line">      <span class="keywordflow">return</span> LEX();</div><div class="line">    }</div><div class="line">  };</div><div class="line">  <span class="keywordtype">int</span> NAMESPACE::LEXER::LEX()</div><div class="line">  {</div><div class="line">    <span class="keyword">static</span> <span class="keyword">const</span> <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> PATTERN_INITIAL(<span class="stringliteral">&quot;(?m)(REGEX)&quot;</span>);</div><div class="line">    <span class="keywordflow">if</span> (!has_matcher())</div><div class="line">    {</div><div class="line">      matcher(<span class="keyword">new</span> Matcher(PATTERN_INITIAL, stdinit(), <span class="keyword">this</span>));</div><div class="line">    }</div><div class="line">    CODE</div><div class="line">    <span class="keywordflow">while</span> (<span class="keyword">true</span>)</div><div class="line">    {</div><div class="line">      <span class="keywordflow">switch</span> (matcher().scan())</div><div class="line">      {</div><div class="line">        <span class="keywordflow">case</span> 0:</div><div class="line">          <span class="keywordflow">if</span> (matcher().at_end())</div><div class="line">          {</div><div class="line">            <span class="keywordflow">return</span> 0;</div><div class="line">          }</div><div class="line">          <span class="keywordflow">else</span></div><div class="line">          {</div><div class="line">            out().put(matcher().input());</div><div class="line">          }</div><div class="line">          <span class="keywordflow">break</span>;</div><div class="line">        <span class="keywordflow">case</span> 1:</div><div class="line">          ACTION</div><div class="line">          <span class="keywordflow">break</span>;</div><div class="line">      }</div><div class="line">    }</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --><p>The Lexer class produced with option <code>−−flex</code> is compatible with Flex (assuming Flex with option <code>-+</code> for C++):</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="flexlexer_8h.html">reflex/flexlexer.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">namespace </span>NAMESPACE {</div><div class="line"></div><div class="line">  <span class="keyword">typedef</span> <a class="code" href="classreflex_1_1_flex_lexer.html">reflex::FlexLexer&lt;reflex::Matcher&gt;</a> FlexLexer;</div><div class="line"></div><div class="line">  <span class="keyword">class </span>LEXER : <span class="keyword">public</span> FlexLexer {</div><div class="line">    MEMBERS</div><div class="line">   <span class="keyword">public</span>:</div><div class="line">    LEXER(</div><div class="line">        CTORARG,</div><div class="line">        <span class="keyword">const</span> <a class="code" href="classreflex_1_1_input.html">reflex::Input</a>&amp; input = <a class="code" href="classreflex_1_1_input.html">reflex::Input</a>(),</div><div class="line">        std::ostream        *os    = NULL)</div><div class="line">      :</div><div class="line">        FlexLexer(input, os)</div><div class="line">    {</div><div class="line">      INIT</div><div class="line">    }</div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">int</span> LEX(<span class="keywordtype">void</span>);</div><div class="line">    <span class="keywordtype">int</span> LEX(</div><div class="line">        <span class="keyword">const</span> <a class="code" href="classreflex_1_1_input.html">reflex::Input</a>&amp; input,</div><div class="line">        std::ostream        *os = NULL)</div><div class="line">    {</div><div class="line">      in(input);</div><div class="line">      <span class="keywordflow">if</span> (os)</div><div class="line">        out(*os);</div><div class="line">      <span class="keywordflow">return</span> LEX();</div><div class="line">    }</div><div class="line">  };</div><div class="line"></div><div class="line">  <span class="keywordtype">int</span> NAMESPACE::LEXER::LEX()</div><div class="line">  {</div><div class="line">    <span class="keyword">static</span> <span class="keyword">const</span> <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> PATTERN_INITIAL(<span class="stringliteral">&quot;(?m)(REGEX)&quot;</span>);</div><div class="line">    <span class="keywordflow">if</span> (!has_matcher())</div><div class="line">    {</div><div class="line">      matcher(<span class="keyword">new</span> Matcher(PATTERN_INITIAL, stdinit(), <span class="keyword">this</span>));</div><div class="line">      <a class="code" href="flexlexer_8h.html#a231e56d8de12d0283bd71ed4e7592171">YY_USER_INIT</a></div><div class="line">    }</div><div class="line">    CODE</div><div class="line">    <span class="keywordflow">while</span> (<span class="keyword">true</span>)</div><div class="line">    {</div><div class="line">      <span class="keywordflow">switch</span> (matcher().scan())</div><div class="line">      {</div><div class="line">        <span class="keywordflow">case</span> 0:</div><div class="line">          <span class="keywordflow">if</span> (matcher().at_end())</div><div class="line">          {</div><div class="line">            <span class="keywordflow">return</span> 0;</div><div class="line">          }</div><div class="line">          <span class="keywordflow">else</span></div><div class="line">          {</div><div class="line">            output(matcher().input());</div><div class="line">          }</div><div class="line">          <a class="code" href="flexlexer_8h.html#a3cc40a460ad7df816678bcc05241e84c">YY_BREAK</a></div><div class="line">        <span class="keywordflow">case</span> 1:</div><div class="line">          <a class="code" href="flexlexer_8h.html#a6198b2fcf96178b24ad4efff2a3debb0">YY_USER_ACTION</a></div><div class="line">          ACTION</div><div class="line">          <a class="code" href="flexlexer_8h.html#a3cc40a460ad7df816678bcc05241e84c">YY_BREAK</a></div><div class="line">      }</div><div class="line">    }</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --><p>To use a custom lexer class that inherits the generated base Lexer class, use option <code>−−class=NAME</code> to declare the name of your custom lexer class (or option <code>−−yyclass=NAME</code> to also enable <code>−−flex</code> compatibility with the <code>yyFlexLexer</code> class). For details, see <a class="el" href="index.html#reflex-inherit">Inheriting Lexer/yyFlexLexer </a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-inherit"></a>
Inheriting Lexer/yyFlexLexer                                  </h2>
<p>To define a custom lexer class that inherits the generated Lexer or the yyFlexLexer class, use option <code>−−class=NAME</code> or option <code>−−yyclass=NAME</code>, respectively. Note that <code>−−yyclass=NAME</code> also enables option <code>−−flex</code> and therefore enables Flex specification syntax.</p>
<p>When a <code>−−class=NAME</code> or <code>−−yyclass=NAME</code> option is specified with the name of your custom lexer class, <b><code>reflex</code></b> generates the <code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> (or <code>yylex()</code>) method code for your custom lexer class. The custom lexer class should declare a public <code>int <a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> method (or <code>int yylex()</code> method with option <code>−−yyclass=NAME</code>). Otherwise, C++ compilation of your custom class will fail.</p>
<p>For example, the following bare-bones custom Lexer class definition simply inherits Lexer and declares a public <code>int <a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> method:</p>
<div class="alt"> <div class="fragment"><div class="line">%option <span class="keyword">class</span>=MyLexer</div><div class="line"></div><div class="line">%{</div><div class="line">  <span class="keyword">class </span>MyLexer : <span class="keyword">public</span> Lexer {</div><div class="line">   <span class="keyword">public</span>:</div><div class="line">    <span class="keywordtype">int</span> <a class="code" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805">lex</a>();</div><div class="line">  };</div><div class="line">%}</div></div><!-- fragment --> </div><p>The <code>int <a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">MyLexer::lex()</a></code> method code is generated by <b><code>reflex</code></b> for this lexer specification.</p>
<p>Options <code>−−lexer=NAME</code> and <code>−−lex=NAME</code> may be combined with <code>−−class=NAME</code> to change the name of the inherited Lexer class and change the name of the <code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> method, respectively.</p>
<p>When using option <code>−−yyclass=NAME</code> the inherited lexer is <code>yyFlexLexer</code>. The custom lexer class should declare a public <code>yylex()</code> method similar to Flex. For example:</p>
<div class="alt"> <div class="fragment"><div class="line">%option yyclass=<span class="stringliteral">&quot;MyLexer&quot;</span></div><div class="line"></div><div class="line">%{</div><div class="line">  <span class="keyword">class </span>MyLexer : <span class="keyword">public</span> yyFlexLexer {</div><div class="line">   <span class="keyword">public</span>:</div><div class="line">    <span class="keywordtype">int</span> yylex();</div><div class="line">  };</div><div class="line">%}</div></div><!-- fragment --> </div><p>The <code>int MyLexer::yylex()</code> method code is generated by <b><code>reflex</code></b> for this lexer specification.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-multiple"></a>
Combining multiple lexers                                    </h2>
<p>To combine multiple lexers in one application, use the <code>−−lexer=NAME</code> option. This option renames the generated lexer class to avoid lexer class name clashes. Use this option in combination with option <code>−−header-file</code> to output a header file with the lexer class declaration to include in your application source code.</p>
<p>Use option <code>−−prefix=NAME</code> to output the generated code in file <em><code>lex.NAME.cpp</code></em> instead of the standard <em><code>lex.yy.cpp</code></em> to avoid file name clashes. This option also affect the <code>−−flex</code> option by generating <code>xxFlexLexer</code> with a <code>xxlex()</code> method when option <code>−−prefix=xx</code> is specified. The generated <code>#define</code> names (some of which are added to support option <code>−−flex</code>) are prefixed to avoid macro name clashes.</p>
<p>Alternatively to <code>−−lexer=NAME</code> you can use <code>−−namespace=NAME</code> to place the generated lexer class in a C++ namespace to avoid lexer class name clashes.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-input"></a>
Switching input sources                                         </h2>
<p>To create a Lexer class instance that reads from a designated input source instead of standard input, pass the input source as the first argument to its constructor and use the second argument to optionally set an <code>std::ostream</code> that is assigned to <code>out()</code> and is used by <code>echo()</code> (likewise, assigned to <code>*yyout</code> and used by <code>ECHO</code> when option <code>−−flex</code> is specified):</p>
<div class="fragment"><div class="line">Lexer lexer(input, std::cout);</div></div><!-- fragment --><p>likewise, with option <code>−−flex</code>:</p>
<div class="fragment"><div class="line">yyFlexLexer lexer(input, std::cout);</div></div><!-- fragment --><p>where <code>input</code> is a <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object. The <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> constructor takes a <code>FILE*</code> descriptor, <code>std::istream</code>, a string <code>std::string</code> or <code>const char*</code>, or a wide string <code>std::wstring</code> or <code>const wchar_t*</code>.</p>
<p>The following methods are available to specify an input source:</p>
<table class="doxtable">
<tr>
<th>RE/flex action </th><th>Flex action </th><th>Result  </th></tr>
<tr>
<td><code>in()</code> </td><td><code>*yyin</code> </td><td>get pointer to current <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> </td></tr>
<tr>
<td><code>in() = i</code> </td><td><code>yyin = &amp;i</code> </td><td>set input <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> </td></tr>
<tr>
<td><code>in(i)</code> </td><td><code><a class="el" href="flexlexer_8h.html#add6f1ba7912f4e259567ebdc120bff17" title="Flex-compatible macro: restart from the given input source. ">yyrestart(i)</a></code> </td><td>reset and scan input from <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> </td></tr>
<tr>
<td><code>in(s)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a3d2545b35aec7c35e6939755777e2a58" title="Flex-compatible macro: scan a string. ">yy_scan_string(s)</a></code> </td><td>reset and scan string <code>s</code> (<code>std::string</code> or <code>char*</code>) </td></tr>
<tr>
<td><code>in(s)</code> </td><td><code><a class="el" href="flexlexer_8h.html#acaf921bfaed6dc25af86fae7ceadc946">yy_scan_wstring(s)</a></code> </td><td>reset and scan wide string <code>s</code> (<code>std::wstring</code> or <code>wchar_t*</code>) </td></tr>
<tr>
<td><code>in(b, n)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a8cef4ba504e54ab2cf998d34ffe047cf">yy_scan_bytes(b, n)</a></code> </td><td>reset and scan <code>n</code> bytes at <code>b</code> address (buffered) </td></tr>
<tr>
<td><code>buffer(b, n+1)</code> </td><td><code>yy_scan_buffer(b, n+2)</code> </td><td>reset and scan <code>n</code> bytes at <code>b</code> address (zero copy) </td></tr>
</table>
<p>For example, to switch input to another source while using the scanner, use <code>in(i)</code> with <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> as an argument:</p>
<div class="fragment"><div class="line"><span class="comment">// read from a file, this also decodes UTF-8/16/32 encodings automatically</span></div><div class="line">FILE *fd = fopen(<span class="stringliteral">&quot;cow.txt&quot;</span>, <span class="stringliteral">&quot;r&quot;</span>);</div><div class="line"><span class="keywordflow">if</span> (fd == NULL)</div><div class="line">  ... <span class="comment">// error, bail out</span></div><div class="line">lexer.in(fd);</div><div class="line">lexer.lex();</div><div class="line"></div><div class="line"><span class="comment">// read from a stream (ASCII or UTF-8)</span></div><div class="line">std::istream i = std::ifstream(<span class="stringliteral">&quot;file&quot;</span>, std::ios::in);</div><div class="line">lexer.in(i);</div><div class="line">lexer.lex();</div><div class="line"></div><div class="line"><span class="comment">// read from a string (0-terminated, ASCII or UTF-8)</span></div><div class="line">lexer.in(<span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line">lexer.lex();</div><div class="line"></div><div class="line"><span class="comment">// read from a memory segment (raw bytes, ASCII, or UTF-8)</span></div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *ptr = ...; <span class="comment">// points to segment</span></div><div class="line"><span class="keywordtype">size_t</span> len = ...;      <span class="comment">// length of the memory segment</span></div><div class="line">lexer.in(ptr, len);</div><div class="line">lexer.lex();</div><div class="line"></div><div class="line"><span class="comment">// read from a wide string, 0-terminated, encoding it to UTF-8 for matching</span></div><div class="line">lexer.in(L<span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line">lexer.lex();</div></div><!-- fragment --><p>You can assign new input with <code>in() = i</code>, which does not reset the lexer's matcher. This means that when the end of the input (EOF) is reached, and you want to switch to new input, then you should clear the EOF state first with <code>lexer.matcher().set_end(false)</code> to reset EOF. Or use <code>lexer.matcher().reset()</code> to clear the state.</p>
<p>Invoking <code>in(i)</code> resets the lexer's matcher (i.e. internally with <code>matcher.reset()</code>). This clears the line and column counters, resets the internal anchor and boundary flags for anchor and word boundary matching, and resets the matcher to consume buffered input.</p>
<p>These <code>in(i)</code> operations specify strings and bytes that are copied to an internal buffer. This is desirable, because the scanner uses a matcher that initializes a buffer, block-wise copies more input to this internal buffer on demand, and modifies this buffered content, e.g. to allow <code>text()</code> to return a 0-terminated <code>char</code> string. Zero copy overhead is obtained with lexer method <code>buffer(b, n)</code> to assign an external buffer:</p>
<div class="fragment"><div class="line"><span class="comment">// read a 0-terminated buffer in place, buffer content is changed!!</span></div><div class="line"><span class="keywordtype">char</span> *base = ...;  <span class="comment">// points to 0-terminated buffer</span></div><div class="line"><span class="keywordtype">size_t</span> size = ...; <span class="comment">// length of the buffer including final \0 byte</span></div><div class="line">lexer.buffer(base, size);</div><div class="line">lexer.lex();</div></div><!-- fragment --><dl class="section warning"><dt>Warning</dt><dd>Function <code>buffer(b, n)</code> scans <code>n</code>-1 bytes at address <code>b</code>. The length <code>n</code> should include the final zero byte at the end of the string.</dd></dl>
<p>With options <code>−−flex</code> and <code>−−bison</code> you can also use classic Flex functions:</p>
<div class="fragment"><div class="line"><span class="comment">// read from a file, this also decodes UTF-8/16/32 encodings automatically</span></div><div class="line">FILE *fd = fopen(<span class="stringliteral">&quot;cow.txt&quot;</span>, <span class="stringliteral">&quot;r&quot;</span>);</div><div class="line"><span class="keywordflow">if</span> (fd == NULL)</div><div class="line">  ... <span class="comment">// error, bail out</span></div><div class="line"><a class="code" href="flexlexer_8h.html#aa789f4617e33fb99594cb04a3688a0c1">yyin</a> = fd;</div><div class="line">yylex();</div><div class="line"></div><div class="line"><span class="comment">// read from a stream (ASCII or UTF-8)</span></div><div class="line">std::istream i = std::ifstream(<span class="stringliteral">&quot;file&quot;</span>, std::ios::in);</div><div class="line"><a class="code" href="flexlexer_8h.html#aa789f4617e33fb99594cb04a3688a0c1">yyin</a> = &amp;i;</div><div class="line">yylex();</div><div class="line"></div><div class="line"><span class="comment">// read from a string (0-terminated, ASCII or UTF-8)</span></div><div class="line"><a class="code" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229">yy_delete_buffer</a>(<a class="code" href="flexlexer_8h.html#aa093d500a6330d06d8e4760c494fac33">YY_CURRENT_BUFFER</a>);</div><div class="line"><a class="code" href="flexlexer_8h.html#a3d2545b35aec7c35e6939755777e2a58">yy_scan_string</a>(<span class="stringliteral">&quot;How now brown cow.&quot;</span>); <span class="comment">// new buffer to scan a string</span></div><div class="line"><span class="comment">// yyin = &quot;How now brown cow.&quot;;       // alternative, does not create a new buffer</span></div><div class="line">yylex();</div><div class="line"></div><div class="line"><span class="comment">// read from a memory segment (raw bytes, ASCII, or UTF-8)</span></div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *ptr = ...; <span class="comment">// points to memory segment</span></div><div class="line"><span class="keywordtype">size_t</span> len = ...;      <span class="comment">// length of the memory segment</span></div><div class="line"><a class="code" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229">yy_delete_buffer</a>(<a class="code" href="flexlexer_8h.html#aa093d500a6330d06d8e4760c494fac33">YY_CURRENT_BUFFER</a>);</div><div class="line"><a class="code" href="flexlexer_8h.html#a8cef4ba504e54ab2cf998d34ffe047cf">yy_scan_bytes</a>(ptr, len); <span class="comment">// new buffer to scan memory</span></div><div class="line">yylex();</div><div class="line"></div><div class="line"><span class="comment">// read from a wide string, 0-terminated, encoding it to UTF-8 for matching</span></div><div class="line"><a class="code" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229">yy_delete_buffer</a>(<a class="code" href="flexlexer_8h.html#aa093d500a6330d06d8e4760c494fac33">YY_CURRENT_BUFFER</a>);</div><div class="line"><a class="code" href="flexlexer_8h.html#acaf921bfaed6dc25af86fae7ceadc946">yy_scan_wstring</a>(L<span class="stringliteral">&quot;How now brown cow.&quot;</span>); <span class="comment">// new buffer to scan a wide string</span></div><div class="line"><span class="comment">// yyin = L&quot;How now brown cow.&quot;;        // alternative, does not create a new buffer</span></div><div class="line">yylex();</div></div><!-- fragment --><p>The <code>yy_scan_string</code>, <code>yy_scan_bytes</code>, and <code>yy_scan_wstring</code> functions create a new buffer (i.e. a new matcher in RE/flex) and replace the old buffer without deleting it. A pointer to the new buffer is returned, which becomes the new <code>YY_CURRENT_BUFFER</code>. You should delete the old buffer with <code><a class="el" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229" title="Flex-compatible macro: delete a buffer. ">yy_delete_buffer(YY_CURRENT_BUFFER)</a></code> before creating a new buffer.</p>
<p>Zero copy overhead is obtained with <code><a class="el" href="flexlexer_8h.html#a0872a059892f77ba25bd253e1ac32d2d">yy_scan_buffer(b, n)</a></code>:</p>
<div class="fragment"><div class="line"><span class="comment">// read a 0-terminated buffer in place, buffer content is changed!!</span></div><div class="line"><span class="keywordtype">char</span> *base = ...;  <span class="comment">// points to 0-terminated buffer</span></div><div class="line"><span class="keywordtype">size_t</span> size = ...; <span class="comment">// length of the buffer including two final \0 bytes</span></div><div class="line"><a class="code" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229">yy_delete_buffer</a>(<a class="code" href="flexlexer_8h.html#aa093d500a6330d06d8e4760c494fac33">YY_CURRENT_BUFFER</a>);</div><div class="line"><a class="code" href="flexlexer_8h.html#a0872a059892f77ba25bd253e1ac32d2d">yy_scan_buffer</a>(base, size);</div><div class="line">yylex();</div></div><!-- fragment --><dl class="section warning"><dt>Warning</dt><dd>The Flex-compatible <code><a class="el" href="flexlexer_8h.html#a0872a059892f77ba25bd253e1ac32d2d">yy_scan_buffer(b, n)</a></code> (when option <code>−−flex</code> is used) scans <code>n</code>-2 bytes at address <code>b</code>. The length <code>n</code> should include <em>two final zero bytes at the end!</em></dd></dl>
<dl class="section note"><dt>Note</dt><dd>Function <code><a class="el" href="flexlexer_8h.html#a0872a059892f77ba25bd253e1ac32d2d">yy_scan_buffer(b, n)</a></code> only touches the first final byte and not the second byte, since this function is the same as calling <code>buffer(b, n-1)</code>. In fact, the specified string may have any final byte value. The final byte of the string will be set to zero when <code>text()</code> (or <code>yytext</code>) or <code>rest()</code> are used. But otherwise the final byte remains completely untouched by the other lexer functions, including <code>echo()</code> (and Flex-compatible <code>ECHO</code>). Only <code>unput(c)</code>, <code>wunput()</code>, <code>text()</code> (or <code>yytext</code>), <code>rest()</code>, and <code>span()</code> modify the buffer contents, where <code>text()</code> and <code>rest()</code> require an extra byte at the end of the buffer to make the strings returned by these functions 0-terminated. This means that you can scan read-only memory of <code>n</code> bytes located at address <code>b</code> by using <code>buffer(b, n+1)</code> safely, for example to read read-only mmap(2) <code>PROT_READ</code> memory, as long as <code>unput(c)</code>,<code>wunput()</code>, <code>text()</code> (or <code>yytext</code>), <code>rest()</code>, and <code>span()</code> are not used.</dd></dl>
<p>The Flex <code>yy_scan_string</code>, <code>yy_scan_bytes</code>, <code>yy_scan_wstring</code>, and <code>yy_scan_buffer</code> functions take an extra last <code>yyscan_t</code> argument for reentrant scanners generated with option <code>−−reentrant</code>, for example:</p>
<div class="fragment"><div class="line"><span class="comment">// read from a file, this also decodes UTF-8/16/32 encodings automatically</span></div><div class="line">FILE *fd = fopen(<span class="stringliteral">&quot;cow.txt&quot;</span>, <span class="stringliteral">&quot;r&quot;</span>);</div><div class="line"><span class="keywordflow">if</span> (fd == NULL)</div><div class="line">  ... <span class="comment">// error, bail out</span></div><div class="line">yyget_in(yyscanner) = fd;</div><div class="line">yylex();</div><div class="line"></div><div class="line"><span class="comment">// read from a stream (ASCII or UTF-8)</span></div><div class="line">std::istream i = std::ifstream(<span class="stringliteral">&quot;file&quot;</span>, std::ios::in);</div><div class="line">yyget_in(yyscanner) = &amp;i;</div><div class="line">yylex();</div><div class="line"></div><div class="line"><span class="comment">// read from a string (0-terminated, ASCII or UTF-8)</span></div><div class="line"><a class="code" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229">yy_delete_buffer</a>(<a class="code" href="flexlexer_8h.html#aa093d500a6330d06d8e4760c494fac33">YY_CURRENT_BUFFER</a>, yyscanner);</div><div class="line"><a class="code" href="flexlexer_8h.html#a3d2545b35aec7c35e6939755777e2a58">yy_scan_string</a>(<span class="stringliteral">&quot;How now brown cow.&quot;</span>, yyscanner); <span class="comment">// new buffer to scan a string</span></div><div class="line">yylex();</div><div class="line"></div><div class="line"><span class="comment">// read from a memory segment (raw bytes, ASCII, or UTF-8)</span></div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *ptr = ...; <span class="comment">// points to memory segment</span></div><div class="line"><span class="keywordtype">size_t</span> len = ...;      <span class="comment">// length of the memory segment</span></div><div class="line"><a class="code" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229">yy_delete_buffer</a>(<a class="code" href="flexlexer_8h.html#aa093d500a6330d06d8e4760c494fac33">YY_CURRENT_BUFFER</a>, yyscanner);</div><div class="line"><a class="code" href="flexlexer_8h.html#a8cef4ba504e54ab2cf998d34ffe047cf">yy_scan_bytes</a>(ptr, len, yyscanner); <span class="comment">// new buffer to scan memory</span></div><div class="line">yylex();</div><div class="line"></div><div class="line"><span class="comment">// read from a wide string, 0-terminated, encoding it to UTF-8 for matching</span></div><div class="line"><a class="code" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229">yy_delete_buffer</a>(<a class="code" href="flexlexer_8h.html#aa093d500a6330d06d8e4760c494fac33">YY_CURRENT_BUFFER</a>, yyscanner);</div><div class="line"><a class="code" href="flexlexer_8h.html#acaf921bfaed6dc25af86fae7ceadc946">yy_scan_wstring</a>(L<span class="stringliteral">&quot;How now brown cow.&quot;</span>, yyscanner); <span class="comment">// new buffer to scan a wide string</span></div><div class="line">yylex();</div><div class="line"></div><div class="line"><span class="comment">// read a 0-terminated buffer in place, buffer content is changed!!</span></div><div class="line"><span class="keywordtype">char</span> *base = ...;  <span class="comment">// points to 0-terminated buffer</span></div><div class="line"><span class="keywordtype">size_t</span> size = ...; <span class="comment">// length of the buffer including final 0 byte</span></div><div class="line"><a class="code" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229">yy_delete_buffer</a>(<a class="code" href="flexlexer_8h.html#aa093d500a6330d06d8e4760c494fac33">YY_CURRENT_BUFFER</a>, yyscanner);</div><div class="line"><a class="code" href="flexlexer_8h.html#a0872a059892f77ba25bd253e1ac32d2d">yy_scan_buffer</a>(base, size, yyscanner);</div><div class="line">yylex();</div></div><!-- fragment --><p>The <code>yyscanner</code> macro is essentially the same is the <code>this</code> pointer that can only be used in lexer methods and in lexer rules. Outside the scope of lexer methods a pointer to your <code>yyFlexLexer lexer</code> object should be used instead, for example <code>yyget_in(&amp;lexer)</code>. Also <code>YY_CURRENT_BUFFER</code> should be replaced by <code>yyget_current_buffer(&amp;lexer)</code>. See also <a class="el" href="index.html#reflex-reentrant">Reentrant scanners</a>.</p>
<p>Switching input before the end of the input source is reached discards all remaining input from that source. To switch input without affecting the current input source, switch matchers instead. The matchers buffer the input and manage the input state, in addition to pattern matching the input.</p>
<p>The following methods are available to specify a matcher <code>Matcher m</code> (a Flex "buffer") for a lexer:</p>
<table class="doxtable">
<tr>
<th>RE/flex action </th><th>Flex action </th><th>Result  </th></tr>
<tr>
<td><code>matcher(m)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a97d5ac1e34fadb1c7121a3bb9837a973" title="Flex-compatible macro: switch to another buffer. ">yy_switch_to_buffer(m)</a></code> </td><td>use matcher <code>m</code> </td></tr>
<tr>
<td><code>new_matcher(i)</code> </td><td><code><a class="el" href="flexlexer_8h.html#aa28e58c21d4fbe7132b6de8edf76e43a" title="Flex-compatible macro: create and return a new buffer. ">yy_create_buffer(i, n)</a></code> </td><td>returns new matcher for <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> </td></tr>
<tr>
<td><code>del_matcher(m)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229" title="Flex-compatible macro: delete a buffer. ">yy_delete_buffer(m)</a></code> </td><td>delete matcher <code>m</code> </td></tr>
<tr>
<td><code>push_matcher(m)</code> </td><td><code><a class="el" href="flexlexer_8h.html#a3807e0a231906457045e429d6721f291" title="Flex-compatible macro: push the current buffer on the stack to use the given buffer. ">yypush_buffer_state(m)</a></code> </td><td>push current matcher, use <code>m</code> </td></tr>
<tr>
<td><code>pop_matcher()</code> </td><td><code><a class="el" href="flexlexer_8h.html#ad0d14a86a4c2feaa7e9e73fd26239dc1" title="Flex-compatible macro: pop buffer from the stack and delete the current buffer. ">yypop_buffer_state()</a></code> </td><td>pop matcher and delete current </td></tr>
<tr>
<td><code>ptr_matcher()</code> </td><td><code>YY_CURRENT_BUFFER</code> </td><td>pointer to current matcher </td></tr>
<tr>
<td><code>has_matcher()</code> </td><td><code>YY_CURRENT_BUFFER != 0</code> </td><td>current matcher is usable </td></tr>
</table>
<p>For example, to switch to a matcher that scans from a new input source, then restores the old input source:</p>
<div class="fragment"><div class="line">... <span class="comment">// scanning etc.</span></div><div class="line">Matcher *oldmatcher = matcher();</div><div class="line">Matcher *newmatcher = new_matcher(input);</div><div class="line">matcher(newmatcher);</div><div class="line">... <span class="comment">// scan the new input</span></div><div class="line">del_matcher(newmatcher);</div><div class="line">matcher(oldmatcher);</div><div class="line">... <span class="comment">// continue scanning the old input</span></div></div><!-- fragment --><p>the same with the <code>−−flex</code> option becomes:</p>
<div class="fragment"><div class="line">... <span class="comment">// scanning etc.</span></div><div class="line"><a class="code" href="flexlexer_8h.html#af8ef002da9f9ba35951176ec0be7ab23">YY_BUFFER_STATE</a> oldbuf = <a class="code" href="flexlexer_8h.html#aa093d500a6330d06d8e4760c494fac33">YY_CURRENT_BUFFER</a>;</div><div class="line"><a class="code" href="flexlexer_8h.html#af8ef002da9f9ba35951176ec0be7ab23">YY_BUFFER_STATE</a> newbuf = <a class="code" href="flexlexer_8h.html#aa28e58c21d4fbe7132b6de8edf76e43a">yy_create_buffer</a>(input, <a class="code" href="flexlexer_8h.html#ae7e51116e747d3390e7a6cfc6532834c">YY_BUF_SIZE</a>);</div><div class="line"><a class="code" href="flexlexer_8h.html#a97d5ac1e34fadb1c7121a3bb9837a973">yy_switch_to_buffer</a>(newbuf);</div><div class="line">... <span class="comment">// scan the new input</span></div><div class="line"><a class="code" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229">yy_delete_buffer</a>(newbuf);</div><div class="line"><a class="code" href="flexlexer_8h.html#a97d5ac1e34fadb1c7121a3bb9837a973">yy_switch_to_buffer</a>(oldbuf);</div><div class="line">... <span class="comment">// continue scanning the old input</span></div></div><!-- fragment --><p>This switches the scanner's input by switching to another matcher. Note that <code>matcher(m)</code> may be used by the virtual <code>wrap()</code> method (or <code>yywrap()</code> when option <code>−−flex</code> is specified) if you use input wrapping after EOF to set things up for continued scanning.</p>
<p>Switching input sources (via either <code>matcher(m)</code>, <code>in(i)</code>, or the Flex functions) does not change the current start condition state.</p>
<p>When the scanner reaches the end of the input, it will check the <code>int wrap()</code> method to detetermine if scanning should continue. If <code>wrap()</code> returns one (1) the scanner terminates and returns zero to its caller. If <code>wrap()</code> returns zero (0) then the scanner continues. In this case <code>wrap()</code> should set up a new input source to scan.</p>
<p>For example, continuing reading from <code>std:cin</code> after some other input source reached EOF:</p>
<div class="alt"> <div class="fragment"><div class="line">%<span class="keyword">class</span>{</div><div class="line">  <span class="keyword">virtual</span> <span class="keywordtype">int</span> wrap() <span class="comment">// note: yywrap() with option −−flex</span></div><div class="line">  {</div><div class="line">    in(std::cin);</div><div class="line">    <span class="keywordflow">return</span> in().good() ? 0 : 1;</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --> </div><p>To implement a <code>wrap()</code> (and <code>yywrap()</code> when option <code>−−flex</code> is specified) in a derived lexer class with option <code>class=NAME</code> (or <code>yyclass=NAME</code>), override the <code>wrap()</code> (or <code>yywrap()</code>) method as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">%option <span class="keyword">class</span>=Tokenizer</div><div class="line"></div><div class="line">%{</div><div class="line">  <span class="keyword">class </span>Tokenizer : Lexer { <span class="comment">// note: yyFlexLexer with option −−flex</span></div><div class="line">   <span class="keyword">public</span>:</div><div class="line">    <span class="keyword">virtual</span> <span class="keywordtype">int</span> wrap() <span class="comment">// note: yywrap() with option −−flex</span></div><div class="line">    {</div><div class="line">      in(std::cin);</div><div class="line">      <span class="keywordflow">return</span> in().good() ? 0 : 1;</div><div class="line">    }</div><div class="line">  };</div><div class="line">%}</div></div><!-- fragment --> </div><p>You can override the <code>wrap()</code> method to set up a new input source when the current input is exhausted. Do not use <code>matcher().input(i)</code> to set a new input source <code>i</code>, because that resets the internal matcher state.</p>
<p>With the <code>−−flex</code> options your can override the <code>yyFlexLexer::yywrap()</code> method that returns an integer 0 (more input available) or 1 (we're done).</p>
<p>With the <code>−−flex</code> and <code>−−bison</code> options you should define a global <code>yywrap()</code> function that returns an integer 0 (more input available) or 1 (we're done).</p>
<p>To set the current input as interactive, such as input from a console, use <code>matcher().interactive()</code> (<code><a class="el" href="flexlexer_8h.html#a1584c58cbc37bb021cbce23ddb7594c7" title="Flex-compatible macro: interactive mode on/off (use only when scanner has started). ">yy_set_interactive(1)</a></code> with option <code>−−flex</code>). This disables buffering of the input and makes the scanner responsive to direct input.</p>
<p>To read from the input without pattern matching, use <code>matcher().input()</code> to read one character at a time (8-bit, ASCII or UTF-8). This function returns EOF if the end of the input was reached. But be careful, the Flex <code><a class="el" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a" title="Flex-compatible macro: read one 8-bit character, returns zero when EOF. ">yyinput()</a></code> and <code>input()</code> functions return 0 instead of an <code>EOF</code> (-1)!</p>
<p>To put back one character unto the input stream, use <code>matcher().unput(c)</code> (or <code>unput(c)</code> with option <code>−−flex</code>) to put byte <code>c</code> back in the input or <code>matcher().wunput(c)</code> to put a (wide) character <code>c</code> back in the input.</p>
<dl class="section warning"><dt>Warning</dt><dd>Functions <code>unput()</code> and <code>wunput()</code> invalidate the previous <code>text()</code> and <code>yytext</code> pointers. Basically, <code>text()</code> and <code>yytext</code> cannot be used after <code>unput()</code>.</dd></dl>
<p>For example, to crudily scan a C/C++ multiline comment we can use the rule:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="stringliteral">&quot;/*&quot;</span>    {  <span class="comment">/* skip multiline comments */</span></div><div class="line">  <span class="keywordtype">int</span> c;</div><div class="line">  <span class="keywordflow">while</span> ((c = <a class="code" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a">yyinput</a>()) != 0)</div><div class="line">  {</div><div class="line">    <span class="keywordflow">if</span> (c == <span class="charliteral">&#39;\n&#39;</span>)</div><div class="line">      ++mylineno;</div><div class="line">    <span class="keywordflow">else</span> <span class="keywordflow">if</span> (c == <span class="charliteral">&#39;*&#39;</span>)</div><div class="line">    {</div><div class="line">      <span class="keywordflow">if</span> ((c = <a class="code" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a">yyinput</a>()) == <span class="charliteral">&#39;/&#39;</span>)</div><div class="line">        <span class="keywordflow">break</span>;</div><div class="line">      unput(c);</div><div class="line">    }</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --> </div><p>We actually do not need to keep track of line numbers explicitly, because <code><a class="el" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a" title="Flex-compatible macro: read one 8-bit character, returns zero when EOF. ">yyinput()</a></code> with RE/flex implicitly updates line numbers, unlike Flex from which this example originates.</p>
<p>Instead of the crude approach shown above, a better alternative is to use a regex <code>/\*.*?\*/</code> or perhaps use start condition states, see <a class="el" href="index.html#reflex-states">Start condition states </a>.</p>
<p>A simpler and faster approach is to use <code>skip("*/")</code> to skip comments:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="stringliteral">&quot;/*&quot;</span>    skip(<span class="stringliteral">&quot;*/&quot;</span>);</div></div><!-- fragment --> </div><p>Using <code>skip()</code> is fast and flushes the internal buffer when searching, unlike <code><a class="el" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a" title="Flex-compatible macro: read one 8-bit character, returns zero when EOF. ">yyinput()</a></code> that maintains the buffer contents to keep <code>text()</code> (and <code>yytext</code>) unchanged.</p>
<p>To grab the rest of the input as a string, use <code>matcher().rest()</code> which returns a <code>const char*</code> string that points to the internal buffer that is enlarged to contain all remaining input. Copy the string before using the matcher again.</p>
<p>To read a number of bytes <code>n</code> into a string buffer <code>s[0..n-1]</code>, use <code>matcher().in.get(s, n)</code>, which is the same as invoking the virtual method <code>matcher().get(s, n)</code>. This matcher method can be overriden by a derived matcher class (to customize reading).</p>
<p>The Flex <code>YY_INPUT</code> macro is not supported by RE/flex. It is recommended to use <code>YY_BUFFER_STATE</code> (Flex), which is a <code><a class="el" href="classreflex_1_1_flex_lexer_1_1_matcher.html" title="Extends reflex::AbstractLexer&lt;M&gt;::Matcher for Flex-compatibility. ">reflex::FlexLexer::Matcher</a></code> class in RE/flex that holds the matcher state and the state of the current input, including the line and column number positions (so unlike Flex, <code>yylineno</code> does not have to be saved and restored when switching buffers). See also section <a class="el" href="index.html#reflex-spec">Lexer specifications </a> on the actions to use.</p>
<p>To implement a custom input handler you can use a proper object-oriented approach: create a derived class of <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> (or <code><a class="el" href="classreflex_1_1_boost_posix_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost POSIX regex matching. ">reflex::BoostPosixMatcher</a></code>) and in the derived class override the <code>size_t <a class="el" href="classreflex_1_1_abstract_matcher.html#a92391ce2c5eab5a6dbce067a8326702e" title="Returns more input directly from the source (method can be overriden, as by reflex::FlexLexer::get(s...">reflex::Matcher::get(char *s, size_t n)</a></code> method for input handling. This function is called with a string buffer <code>s</code> of size <code>n</code> bytes. Fill the string buffer <code>s</code> up to <code>n</code> bytes and return the number of bytes stored in <code>s</code>. Return zero upon EOF. Use <b><code>reflex</code></b> options <code>−−matcher=NAME</code> and <code>−−pattern=<a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> to use your new matcher class <code>NAME</code> (or leave out <code>−−pattern</code> for Boost.Regex derived matchers).</p>
<p>The <code>FlexLexer</code> lexer class that is the base class of the <code>yyFlexLexer</code> lexer class generated with <b><code>reflex</code></b> option <code>−−flex</code> defines a virtual <code>size_t LexerInput(char*, size_t)</code> method. This approach is compatible with Flex. The virtual method can be redefined in the generated <code>yyFlexLexer</code> lexer to consume input from some source of text:</p>
<div class="alt"> <div class="fragment"><div class="line">%<span class="keyword">class</span>{</div><div class="line">  <span class="keyword">virtual</span> <span class="keywordtype">size_t</span> LexerInput(<span class="keywordtype">char</span> *s, <span class="keywordtype">size_t</span> n)</div><div class="line">  {</div><div class="line">    <span class="keywordtype">size_t</span> k;</div><div class="line">    <span class="comment">// populate s[0..k-1] for some k with k &lt;= n</span></div><div class="line">    <span class="keywordflow">return</span> k; <span class="comment">// return number of bytes filled in s[]</span></div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --> </div><p>The <code>LexerInput</code> method may be invoked multiple times by the matcher engine and should eventually return zero to indicate the end of input is reached (e.g. when at EOF).</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-multiple-input"></a>
Multiple input sources                                 </h2>
<p>A typical scenario for a compiler of a programming language is to process <code>include</code> directives in the source input that should include the source of another file before continuing with the current input.</p>
<p>For example, the following specification defines a lexer that processes <code>#include</code> directives by switching matchers and using the stack of matchers to permit nested <code>#include</code> directives up to a depth of as much as 99 files:</p>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &lt;stdio.h&gt;</span></div><div class="line">}</div><div class="line"></div><div class="line">%<span class="keyword">class</span>{</div><div class="line"></div><div class="line">  <span class="keywordtype">int</span> depth;</div><div class="line"></div><div class="line">  <span class="keywordtype">void</span> include_file()</div><div class="line">  {</div><div class="line">    depth++;</div><div class="line">    <span class="keywordflow">if</span> (depth &gt; 99)</div><div class="line">      exit(EXIT_FAILURE);           <span class="comment">// max include depth exceeded</span></div><div class="line">    <span class="keywordtype">char</span> *q = strchr(text(), <span class="charliteral">&#39;&quot;&#39;</span>);  <span class="comment">// get ...&quot;filename&quot;</span></div><div class="line">    <span class="keywordtype">char</span> *file = strdup(q + 1);     <span class="comment">// get filename&quot;</span></div><div class="line">    file[strlen(file) - 1] = <span class="charliteral">&#39;\0&#39;</span>;  <span class="comment">// get filename</span></div><div class="line">    FILE *fd = fopen(file, <span class="stringliteral">&quot;r&quot;</span>);</div><div class="line">    free(file);</div><div class="line">    <span class="keywordflow">if</span> (!fd)</div><div class="line">      exit(EXIT_FAILURE);           <span class="comment">// cannot open file</span></div><div class="line">    push_matcher(new_matcher(fd));  <span class="comment">// push current matcher, use new matcher</span></div><div class="line">  }</div><div class="line"></div><div class="line">  <span class="keywordtype">bool</span> end_of_file()</div><div class="line">  {</div><div class="line">    <span class="keywordflow">if</span> (depth == 0)</div><div class="line">      <span class="keywordflow">return</span> <span class="keyword">true</span>;                  <span class="comment">// return true: no more input to read</span></div><div class="line">    fclose(in());                   <span class="comment">// close current input in() (a FILE*)</span></div><div class="line">    pop_matcher();                  <span class="comment">// delete current matcher, pop matcher</span></div><div class="line">    depth--;</div><div class="line">    <span class="keywordflow">return</span> <span class="keyword">false</span>;                   <span class="comment">// return false: continue reading</span></div><div class="line">  }</div><div class="line"></div><div class="line">}</div><div class="line"></div><div class="line">%init{</div><div class="line">  depth = 0;</div><div class="line">}</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">^\h*#include\h*\<span class="stringliteral">&quot;.*?\&quot;    include_file();</span></div><div class="line"><span class="stringliteral">.|\n                      echo();</span></div><div class="line"><span class="stringliteral">&lt;&lt;EOF&gt;&gt;                   if (end_of_file()) return 0;</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">%%</span></div></div><!-- fragment --> </div><p>With option <code>−−flex</code>, the statement <code>push_matcher(new_matcher(fd))</code> above becomes <code><a class="el" href="flexlexer_8h.html#a3807e0a231906457045e429d6721f291" title="Flex-compatible macro: push the current buffer on the stack to use the given buffer. ">yypush_buffer_state(yy_create_buffer(fd, YY_BUF_SIZE))</a></code> and <code>pop_matcher()</code> becomes <code><a class="el" href="flexlexer_8h.html#ad0d14a86a4c2feaa7e9e73fd26239dc1" title="Flex-compatible macro: pop buffer from the stack and delete the current buffer. ">yypop_buffer_state()</a></code>. For comparison, here is a C-based classic Flex example specification that works with RE/flex too:</p>
<div class="alt"> <div class="fragment"><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &lt;stdio.h&gt;</span></div><div class="line">  <span class="keywordtype">int</span> depth = 0;</div><div class="line">%}</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">^[ \t]*#include[ \t]*\<span class="stringliteral">&quot;.*?\&quot;    {</span></div><div class="line"><span class="stringliteral">    depth++;</span></div><div class="line"><span class="stringliteral">    if (depth &gt; 99)</span></div><div class="line"><span class="stringliteral">      exit(EXIT_FAILURE);           // max include depth exceeded</span></div><div class="line"><span class="stringliteral">    char *q = strchr(yytext, &#39;&quot;</span><span class="stringliteral">&#39;);  // get ...&quot;filename&quot;</span></div><div class="line"><span class="stringliteral">    char *file = strdup(q + 1);     // get filename&quot;</span></div><div class="line"><span class="stringliteral">    file[strlen(file) - 1] = &#39;</span>\0<span class="stringliteral">&#39;;  // get filename</span></div><div class="line"><span class="stringliteral">    FILE *fd = fopen(file, &quot;r&quot;);</span></div><div class="line"><span class="stringliteral">    free(file);</span></div><div class="line"><span class="stringliteral">    if (!fd)</span></div><div class="line"><span class="stringliteral">      exit(EXIT_FAILURE);           // cannot open file</span></div><div class="line"><span class="stringliteral">    YY_BUFFER_STATE buf = yy_create_buffer(fd, YY_BUF_SIZE);</span></div><div class="line"><span class="stringliteral">    yypush_buffer_state(buf);       // push current buffer, use new buffer</span></div><div class="line"><span class="stringliteral">}</span></div><div class="line"><span class="stringliteral">.|\n                      ECHO;</span></div><div class="line"><span class="stringliteral">&lt;&lt;EOF&gt;&gt;                   {</span></div><div class="line"><span class="stringliteral">    fclose(yyin);                   // close current input</span></div><div class="line"><span class="stringliteral">    yypop_buffer_state();           // delete current buffer, pop buffer</span></div><div class="line"><span class="stringliteral">    if (!YY_CURRENT_BUFFER)         // no buffer, we&#39;</span>re done</div><div class="line">      <a class="code" href="flexlexer_8h.html#add62fdebca5f1fac074d8c918dc45eec">yyterminate</a>();</div><div class="line">    depth--;</div><div class="line">}</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-states"></a>
Start condition states                                         </h2>
<p>Start conditions are used to group rules and selectively activate rules when the start condition state becomes active.</p>
<p>A rule with a pattern that is prefixed with one ore more start conditions will only be active when the scanner is in one of these start condition states.</p>
<p>For example:</p>
<div class="alt"> <div class="fragment"><div class="line">&lt;A,B&gt;pattern1    action1</div><div class="line">&lt;A&gt;pattern2      action2</div><div class="line">&lt;B&gt;pattern3      action3</div></div><!-- fragment --> </div><p>When the scanner is in state <code>A</code> rules 1 and 2 are active. When the scanner is in state <code>B</code> rules 1 and 3 are active.</p>
<p>Start conditions are declared in <a class="el" href="index.html#reflex-spec-defs">The definitions section</a> (the first section) of the lexer specification using <em><code>%state</code></em> or <em><code>%xstate</code></em> (or <em><code>%s</code></em> and <em><code>%x</code></em> for short) followed by a space-separated list of names called <em>start symbols</em>. Start conditions declared with <code>%s</code> are <em>inclusive start conditions</em>. Start conditions declared with <em><code>%x</code></em> are <em>exclusive start conditions</em>:</p>
<p>If a start condition is inclusive, then all rules without a start condition and rules with the corresponding start condition will be active.</p>
<p>If a start condition is exclusive, only the rules with the corresponding start condition will be active.</p>
<p>When declaring start symbol names it is recommended to use all upper case to avoid name clashes with other Lexer class members. For example, we cannot use <code>text</code> as a start symbol name because <code>text()</code> is a Lexer method. When option <code>−−flex</code> is specified, start symbol names are macros for compatibility with Lex/Flex.</p>
<p>The scanner is initially in the <code>INITIAL</code> start condition state. The <code>INITIAL</code> start condtion is inclusive: all rules without a start condition and those prefixed with the <code>INITIAL</code> start condition are active when the scanner is in the <code>INITIAL</code> start condition state.</p>
<p>The special start condition prefix <code>&lt;*&gt;</code> matches every start condition. The prefix <code>&lt;*&gt;</code> is not needed for <code>&lt;&lt;EOF&gt;&gt;</code> rules, because unprefixed <code>&lt;&lt;EOF&gt;&gt;</code> rules are always active as a special case. The <code>&lt;&lt;EOF&gt;&gt;</code> pattern and this exception were originally introduced by Flex.</p>
<p>For example:</p>
<div class="alt"> <div class="fragment"><div class="line">%s A</div><div class="line">%x X</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">&lt;A,X&gt;pattern1    action1    <span class="comment">// rule for states A and X</span></div><div class="line">&lt;A&gt;pattern2      action2    <span class="comment">// rule for state A</span></div><div class="line">&lt;X&gt;pattern3      action3    <span class="comment">// rule for state X</span></div><div class="line">&lt;*&gt;pattern4      action4    <span class="comment">// rule for states INITIAL, A and X</span></div><div class="line">pattern5         action5    <span class="comment">// rule for states INITIAL and A</span></div><div class="line">&lt;&lt;EOF&gt;&gt;          action6    <span class="comment">// rule for states INITIAL, A and X</span></div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>When the scanner is in state <code>INITIAL</code> rules 4, 5, and 6 are active. When the scanner is in state <code>A</code> rules 1, 2, 4, 5, and 6 are active. When the scanner is in state <code>X</code> rules 1, 3, 4, and 6 are active. Note that <code>A</code> is inclusive whereas <code>X</code> is exclusive.</p>
<p>To switch to a start condition state, use <code>start(START)</code> (or <code>BEGIN START</code> when option <code>−−flex</code> is specified). To get the current state use <code>start()</code> (or <code>YY_START</code> when option <code>−−flex</code> is specified). Switching start condition states in your scanner allows you to create "mini-scanners" to scan portions of the input that are syntactically different from the rest of the input, such as comments:</p>
<div class="alt"> <div class="fragment"><div class="line">%x COMMENT</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line"><span class="stringliteral">&quot;/*&quot;</span>    start(COMMENT);    <span class="comment">// with −−flex use: BEGIN COMMENT;</span></div><div class="line">.|\n    echo();            <span class="comment">// with −−flex use: ECHO;</span></div><div class="line"></div><div class="line">&lt;COMMENT&gt;[^*]*         <span class="comment">// eat anything that is not a &#39;*&#39;</span></div><div class="line">&lt;COMMENT&gt;<span class="stringliteral">&quot;*&quot;</span>+[^*/]*    <span class="comment">// eat &#39;*&#39;s not followed by a &#39;/&#39;</span></div><div class="line">&lt;COMMENT&gt;<span class="stringliteral">&quot;*&quot;</span>+<span class="stringliteral">&quot;/&quot;</span>       start(INITIAL);</div><div class="line">&lt;COMMENT&gt;&lt;&lt;EOF&gt;&gt;       std::cerr &lt;&lt; <span class="stringliteral">&quot;EOF in comment\n&quot;</span>; <span class="keywordflow">return</span> 1;</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>Start symbols are actually integer values, where <code>INITIAL</code> is 0. This means that you can store a start symbol value in a variable. You can also push the current start condition on a stack and transition to start condition <code>START</code> with <code>push_state(START)</code>. To transition to a start condition that is on the top of the stack and pop it use <code>pop_state()</code>. The <code>top_state()</code> returns the start condition that is on the top of the stack:</p>
<div class="alt"> <div class="fragment"><div class="line">%x COMMENT</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line"><span class="stringliteral">&quot;/*&quot;</span>    push_state(COMMENT);</div><div class="line">.|\n    echo();</div><div class="line"></div><div class="line">&lt;COMMENT&gt;[^*]*         <span class="comment">// eat anything that is not a &#39;*&#39;</span></div><div class="line">&lt;COMMENT&gt;<span class="stringliteral">&quot;*&quot;</span>+[^*/]*    <span class="comment">// eat &#39;*&#39;s not followed by a &#39;/&#39;</span></div><div class="line">&lt;COMMENT&gt;<span class="stringliteral">&quot;*&quot;</span>+<span class="stringliteral">&quot;/&quot;</span>       pop_state();</div><div class="line">&lt;COMMENT&gt;&lt;&lt;EOF&gt;&gt;       std::cerr &lt;&lt; <span class="stringliteral">&quot;EOF in comment\n&quot;</span>; <span class="keywordflow">return</span> 1;</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>When many rules are prefixed by the same start conditions, you can simplify the rules by placing them in a <em>start condition scope</em>:</p>
<div class="alt"> <div class="fragment"><div class="line">&lt;COMMENT&gt;{</div><div class="line">[^*]*         <span class="comment">// eat anything that is not a &#39;*&#39;</span></div><div class="line"><span class="stringliteral">&quot;*&quot;</span>+[^*/]*    <span class="comment">// eat &#39;*&#39;s not followed by a &#39;/&#39;</span></div><div class="line"><span class="stringliteral">&quot;*&quot;</span>+<span class="stringliteral">&quot;/&quot;</span>       start(INITIAL);</div><div class="line">&lt;&lt;EOF&gt;&gt;       std::cerr &lt;&lt; <span class="stringliteral">&quot;EOF in comment\n&quot;</span>; <span class="keywordflow">return</span> 1;</div><div class="line">}</div></div><!-- fragment --> </div><p>Start condition scopes may be nested. A nested scope extends the scope of start conditions that will be associated with the rules in the nested scope.</p>
<p>For example:</p>
<div class="alt"> <div class="fragment"><div class="line">%s A</div><div class="line">%x X</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">pattern    action    <span class="comment">// rule for states INITIAL and A</span></div><div class="line"></div><div class="line">&lt;A&gt;{</div><div class="line">pattern    action    <span class="comment">// rule for state A</span></div><div class="line">&lt;X&gt;{</div><div class="line">pattern    action    <span class="comment">// rule for states A and X</span></div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line">&lt;X&gt;{</div><div class="line">pattern    action    <span class="comment">// rule for state X</span></div><div class="line">&lt;A&gt;{</div><div class="line">pattern    action    <span class="comment">// rule for states A and X</span></div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>Designating a start condition as inclusive or exclusive is effective only for rules that are not associated with a start condition scope. That is, inclusive start condition states are implicitly associated with rules unless a rule has a start condition scope that explicitly associates start condition states with the rule.</p>
<p>RE/flex extends the syntax of start conditions scopes beyond Flex syntax, allowing the removal of start conditions from the current scope. A start condition name prefixed with the <code>^</code> operator is removed from the current scope:</p>
<div class="alt"> <div class="fragment"><div class="line">%s A B C</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">&lt;B,C&gt;{</div><div class="line">pattern    action    <span class="comment">// rule for states B and C</span></div><div class="line">&lt;A,^B&gt;{</div><div class="line">pattern    action    <span class="comment">// rule for states A and C</span></div><div class="line">}</div><div class="line">}</div><div class="line"></div><div class="line">&lt;*,^A,^C&gt;pattern    action    <span class="comment">// rule for states INITIAL and B (all states except A and C)</span></div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>Note that scopes should be read from outer to inner scope, and from left to right in a <code>&lt;...&gt;</code> scope declaration. This means that <code>&lt;*,^A,^C&gt;</code> first extends the scope to include all start conditions and then removes <code>A</code> and <code>C</code>.</p>
<p>A start condition cannot be removed when it is not included in the current scope. For example, <code>&lt;*,^A&gt;</code> is correct but <code>&lt;^A,*&gt;</code> is incorrect when used as a top-level scope.</p>
<p>Empty <code>&lt;&gt;</code> without start condition states cannot be specified because this is a valid regex pattern. To remove all states from a scope use <code>&lt;^*&gt;</code>. This construct is only useful when the empty scope is extended by start conditions specified in sub-scopes.</p>
<dl class="section note"><dt>Note</dt><dd>Contrary to some Flex manuals, rules cannot be indented in a start condition scope in Flex and RE/flex. When a code block is specified indented at the begin of a start condition scope it is considered an initial code block, see <a class="el" href="index.html#reflex-code-blocks">Initial code blocks </a>.</dd></dl>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-code-blocks"></a>
Initial code blocks                                       </h2>
<p>An initial code block may be placed at the start of the rules section or in a condition scope. This code block is executed each time the scanner is invoked (i.e. when <code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">lex()</a></code> or <code>yylex()</code> is called) before matching a pattern. Initial code blocks may be associated with start condition states as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">%s A</div><div class="line">%x X</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">%{</div><div class="line">  <span class="comment">// Code block for all inclusive states (INITIAL and A)</span></div><div class="line">%}</div><div class="line">pattern    action    <span class="comment">// rule for states INITIAL and A</span></div><div class="line">...</div><div class="line">pattern    action    <span class="comment">// rule for states INITIAL and A</span></div><div class="line"></div><div class="line">&lt;X&gt;{</div><div class="line">%{</div><div class="line">  <span class="comment">// Code block for state X</span></div><div class="line">%}</div><div class="line">pattern    action    <span class="comment">// rule for state X</span></div><div class="line">...</div><div class="line">pattern    action    <span class="comment">// rule for state X</span></div><div class="line">}</div><div class="line"></div><div class="line">&lt;*&gt;{</div><div class="line">%{</div><div class="line">  <span class="comment">// Code block for all inclusive and exclusive states (INITIAL, A, and X)</span></div><div class="line">%}</div><div class="line">pattern    action    <span class="comment">// rule for states INITIAL, A, and X</span></div><div class="line">...</div><div class="line">pattern    action    <span class="comment">// rule for states INITIAL, A, and X</span></div><div class="line">}</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>Initial code blocks should be indented or should be placed within <em><code>%{ %}</code></em> blocks.</p>
<p>An initial code block can be used to configure the lexer's matcher, since a new matcher with the lexer patterns is created by the lexer just before the rules are matched. For example:</p>
<div class="alt"> <div class="fragment"><div class="line">%<span class="keyword">class</span>{</div><div class="line">  <span class="keywordtype">bool</span> init_matcher;</div><div class="line">}</div><div class="line"></div><div class="line">%init{</div><div class="line">  init_matcher = <span class="keyword">true</span>;</div><div class="line">}</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">%{</div><div class="line">  <span class="keywordflow">if</span> (init_matcher)            <span class="comment">// init the new matcher?</span></div><div class="line">  {</div><div class="line">    init_matcher = <span class="keyword">false</span>;      <span class="comment">// init only once</span></div><div class="line">    <span class="keywordflow">if</span> (BUFFER_ALL)            <span class="comment">// buffer all input at once?</span></div><div class="line">      matcher().buffer();      <span class="comment">// same as %option batch</span></div><div class="line">    <span class="keywordflow">else</span> <span class="keywordflow">if</span> (INTERACTIVE)      <span class="comment">// console-based (TTY) &quot;interactive&quot; input?</span></div><div class="line">      matcher().interactive(); <span class="comment">// same as %option interactive   </span></div><div class="line">    matcher().tabs(4);         <span class="comment">// same as %option tabs=4</span></div><div class="line">  }</div><div class="line">%}</div></div><!-- fragment --> </div><p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-bison"></a>
Interfacing with Bison/Yacc                                     </h2>
<p>The <a href="dinosaur.compilertools.net/#bison">Bison</a> tools generate parsers that invoke the global C function <code>yylex()</code> to get the next token. Tokens are integer values returned by <code>yylex()</code>.</p>
<p>To support Bison parsers use <b><code>reflex</code></b> option <code>−−bison</code>. This option generates a scanner with a global lexer object <code>YY_SCANNER</code> and a global <code>YY_EXTERN_C int yylex()</code> function. When the Bison parser is compiled in C and the scanner is compiled in C++, you must set <code>YY_EXTERN_C</code> in the lex specification to <code>extern "C"</code> to enable C linkage rules:</p>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &quot;y.tab.h&quot;</span>               <span class="comment">/* include y.tab.h generated by bison */</span></div><div class="line"><span class="preprocessor">  #define YY_EXTERN_C extern &quot;C&quot;   </span><span class="comment">/* yylex() must use C linkage rules */</span><span class="preprocessor"></span></div><div class="line">}</div><div class="line"></div><div class="line">%option noyywrap bison</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">[0-9]+               yylval.num = strtol(text(), NULL, 10);</div><div class="line">                     <span class="keywordflow">return</span> CONST_NUMBER;</div><div class="line">\<span class="stringliteral">&quot;([^\\&quot;</span>]|\\<span class="stringliteral">&quot;)*\&quot;    yylval.str = text();</span></div><div class="line"><span class="stringliteral">                     return CONST_STRING;</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">%%</span></div></div><!-- fragment --> </div><p>Note that <code>−−noyywrap</code> may be specified to remove the dependency on the global <code>yywrap()</code> function that is not defined.</p>
<p>This example sets the global <code>yylval.num</code> to the integer scanned or <code>yylval.str</code> to the string scanned. It assumes that the Bison/Yacc grammar file defines the tokens <code>CONST_NUMBER</code> and <code>CONST_STRING</code> and the type <code>YYSTYPE</code> of <code>yylval</code>. For example:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.y file) */</span></div><div class="line"></div><div class="line">%{</div><div class="line">  <span class="keyword">extern</span> <span class="keywordtype">int</span> yylex(<span class="keywordtype">void</span>);</div><div class="line">%}</div><div class="line"></div><div class="line">%<span class="keyword">union </span>{         <span class="comment">// YYSTYPE yylval is a union:</span></div><div class="line">  <span class="keywordtype">int</span> num;       <span class="comment">// yylval.num</span></div><div class="line">  <span class="keywordtype">char</span>* str;     <span class="comment">// yylval.str</span></div><div class="line">}</div><div class="line"></div><div class="line">%token &lt;num&gt; CONST_NUMBER</div><div class="line">%token &lt;str&gt; CONST_STRING</div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div></div><!-- fragment --> </div><p><code>YYSTYPE</code> is a union defined by Bison or you can set it as an option <em><code>%option YYSTYPE=type</code></em> in a lexer specification.</p>
<p>When option <code>−−flex</code> is specified with <code>−−bison</code>, the <code>yytext</code>, <code>yyleng</code>, and <code>yylineno</code> globals are accessible to the Bison/Yacc parser. In fact, all Flex actions and variables are globally accessible (outside <a class="el" href="index.html#reflex-spec-rules">The rules section</a> of the lexer specification) with the exception of <code>yy_push_state</code>, <code>yy_pop_state</code>, and <code>yy_top_state</code> that are class methods. Furthermore, <code>yyin</code> and <code>yyout</code> are macros and cannot be (re)declared or accessed as global variables, but these can be used as if they are variables to assign a new input source and to set the output stream. To avoid compilation errors when using globals such as <code>yyin</code>, use <b><code>reflex</code></b> option <code>−−header-file</code> to generate a header file <em><code>lex.yy.h</code></em> to include in your code. Finally, in code outside of <a class="el" href="index.html#reflex-spec-rules">The rules section</a> you must use <code><a class="el" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a" title="Flex-compatible macro: read one 8-bit character, returns zero when EOF. ">yyinput()</a></code> instead of <code>input()</code>, use the global action <code><a class="el" href="flexlexer_8h.html#a978e5155bc5a5e66b2cadcb7b2124c72" title="Flex-compatible macro: put one 8-bit character back onto the input stream to be read again...">yyunput()</a></code> instead of <code>unput()</code>, and use the global action <code><a class="el" href="flexlexer_8h.html#a01e073defdea723b59e8e4415648ac47" title="Flex-compatible macro: output one 8-bit character. ">yyoutput()</a></code> instead of <code>output()</code>.</p>
<p>See the generated <em><code>lex.yy.cpp</code></em> "BISON" section, which contains declarations specific to Bison when the <code>−−bison</code> option is specified.</p>
<p>There are two approaches for a Bison parser to work with a scanner. Either the Bison/Yacc grammar file should include the externs we need to import from the scanner:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.y file) assuming C with externs defined by the scanner using YY_EXTERN_C */</span></div><div class="line"></div><div class="line">%{</div><div class="line">  <span class="keyword">extern</span> <span class="keywordtype">int</span> yylex(<span class="keywordtype">void</span>);</div><div class="line">  <span class="keyword">extern</span> <span class="keywordtype">char</span> *<a class="code" href="flexlexer_8h.html#a0d71f919dbec1ffd74b2460fa7e5ac28">yytext</a>;</div><div class="line">  <span class="keyword">extern</span> <a class="code" href="flexlexer_8h.html#aa9522b9e313cd74e4e688e358cbca836">yy_size_t</a> <a class="code" href="flexlexer_8h.html#afa07a629486cb790560bb95713ec7794">yyleng</a>;</div><div class="line">  <span class="keyword">extern</span> <span class="keywordtype">int</span> <a class="code" href="flexlexer_8h.html#ad71cf0fddcfe4f61de0929105b33226c">yylineno</a>;</div><div class="line">%}</div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div></div><!-- fragment --> </div><p>or a better approach is to generate a <em><code>lex.yy.h</code></em> header file with option <code>−−header-file</code> and use this header file in the Bison/Yacc grammar file:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.y file) for C++ */</span></div><div class="line"></div><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span></div><div class="line">%}</div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div></div><!-- fragment --> </div><p>The second option requires the generated parser to be compiled in C++, because <em><code>lex.yy.h</code></em> contains C++ declarations.</p>
<dl class="section note"><dt>Note</dt><dd>The ugly Flex macro <code>YY_DECL</code> is not supported by RE/flex. This macro is needed with Flex to redeclare the <code>yylex()</code> function signature, for example to take an additional <code>yylval</code> parameter that must be passed through from <code>yyparse()</code> to <code>yylex()</code>. Because the generated scanner uses a Lexer class for scanning, the class can be extended with <em><code>%class{ }</code></em> block to hold state information and additional token-related values. These values can then be exchanged with the parser using getters and setters, which is preferred over changing the <code>yylex()</code> function signature with <code>YY_DECL</code>.</dd></dl>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-bison-mt-safe"></a>
Bison and thread-safety</h3>
<p>Bison and Yacc are not thread-safe because the generated code uses and updates global variables. Yacc and Bison use the global variable <code>yylval</code> to exchange token values. By contrast, thread-safe reentrant Bison parsers pass the <code>yylval</code> to the <code>yylex()</code> function as a parameter. RE/flex supports all of these Bison-specific features.</p>
<p>The following combinations of options are available to generate scanners for Bison:</p>
<table class="doxtable">
<tr>
<th>Options </th><th>Method </th><th>Global functions and variables  </th></tr>
<tr>
<td>&#160; </td><td><code>int <a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">Lexer::lex()</a></code> </td><td>no global variables, but doesn't work with Bison </td></tr>
<tr>
<td><code>−−flex</code> </td><td><code>int yyFlexLexer::yylex()</code> </td><td>no global variables, but doesn't work with Bison </td></tr>
<tr>
<td><code>−−bison</code> </td><td><code>int <a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">Lexer::lex()</a></code> </td><td><code>Lexer YY_SCANNER</code>, <code>int yylex()</code>, <code>YYSTYPE yylval</code> </td></tr>
<tr>
<td><code>−−flex</code> <code>−−bison</code> </td><td><code>int yyFlexLexer::yylex()</code> </td><td><code>yyFlexLexer YY_SCANNER</code>, <code>int yylex()</code>, <code>YYSTYPE yylval</code>, <code>char *yytext</code>, <code>yy_size_t yyleng</code>, <code>int yylineno</code> </td></tr>
<tr>
<td><code>−−bison</code> <code>−−reentrant</code> </td><td><code>int <a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">Lexer::lex()</a></code> </td><td><code>int yylex(yyscan_t)</code>, <code>void yylex_init(yyscan_t*)</code>, <code>void yylex_destroy(yyscan_t)</code> </td></tr>
<tr>
<td><code>−−flex</code> <code>−−bison</code> <code>−−reentrant</code> </td><td><code>int <a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">yyFlexLexer::lex()</a></code> </td><td><code>int yylex(yyscan_t)</code>, <code>void yylex_init(yyscan_t*)</code>, <code>void yylex_destroy(yyscan_t)</code> </td></tr>
<tr>
<td><code>−−bison-locations</code> </td><td><code>int Lexer::lex(YYSTYPE&amp; yylval)</code> </td><td><code>Lexer YY_SCANNER</code>, <code>int yylex(YYSTYPE *yylval, YYLTYPE *yylloc)</code> </td></tr>
<tr>
<td><code>−−flex</code> <code>−−bison-locations</code> </td><td><code>int yyFlexLexer::yylex(YYSTYPE&amp; yylval)</code> </td><td><code>yyFlexLexer YY_SCANNER</code>, <code>int yylex(YYSTYPE *yylval, YYLTYPE *yylloc)</code> </td></tr>
<tr>
<td><code>−−bison-bridge</code> </td><td><code>int Lexer::lex(YYSTYPE&amp; yylval)</code> </td><td><code>int yylex(YYSTYPE *yylval, yyscan_t)</code>, <code>void yylex_init(yyscan_t*)</code>, <code>void yylex_destroy(yyscan_t)</code> </td></tr>
<tr>
<td><code>−−flex</code> <code>−−bison-bridge</code> </td><td><code>int yyFlexLexer::yylex(YYSTYPE&amp; yylval)</code> </td><td><code>int yylex(YYSTYPE *yylval, yyscan_t)</code>, <code>void yylex_init(yyscan_t*)</code>, <code>void yylex_destroy(yyscan_t)</code> </td></tr>
<tr>
<td><code>−−bison-bridge</code> <code>−−bison-locations</code> </td><td><code>int Lexer::lex(YYSTYPE&amp; yylval)</code> </td><td><code>int yylex(YYSTYPE *yylval, YYLTYPE *yylloc, yyscan_t)</code>, <code>void yylex_init(yyscan_t*)</code>, <code>void yylex_destroy(yyscan_t)</code> </td></tr>
<tr>
<td><code>−−flex</code> <code>−−bison-bridge</code> <code>−−bison-locations</code> </td><td><code>int yyFlexLexer::yylex(YYSTYPE&amp; yylval)</code> </td><td><code>int yylex(YYSTYPE *yylval, YYLTYPE *yylloc, yyscan_t)</code>, <code>void yylex_init(yyscan_t*)</code>, <code>void yylex_destroy(yyscan_t)</code> </td></tr>
<tr>
<td><code>−−bison-cc</code> </td><td><code>int Lexer::yylex(YYSTYPE *yylval)</code> </td><td>no global variables </td></tr>
<tr>
<td><code>−−flex</code> <code>−−bison-cc</code> </td><td><code>int yyFlexLexer::yylex(YYSTYPE *yylval)</code> </td><td>no global variables </td></tr>
<tr>
<td><code>−−bison-cc</code> <code>−−bison-locations</code> </td><td><code>int Lexer::yylex(YYSTYPE *yylval, YYLTYPE *yylloc)</code> </td><td>no global variables </td></tr>
<tr>
<td><code>−−flex</code> <code>−−bison-cc</code> <code>−−bison-locations</code> </td><td><code>int yyFlexLexer::yylex(YYSTYPE *yylval, YYLTYPE *yylloc)</code> </td><td>no global variables </td></tr>
<tr>
<td><code>−−bison-complete</code> </td><td><code>PARSER::symbol_type Lexer::yylex()</code> </td><td>no global variables </td></tr>
<tr>
<td><code>−−flex</code> <code>−−bison-complete</code> </td><td><code>PARSER::symbol_type yyFlexLexer::yylex()</code> </td><td>no global variables </td></tr>
<tr>
<td><code>−−bison-complete</code> <code>−−bison-locations</code> </td><td><code>PARSER::symbol_type Lexer::yylex()</code> </td><td>no global variables </td></tr>
<tr>
<td><code>−−flex</code> <code>−−bison-complete</code> <code>−−bison-locations</code> </td><td><code>PARSER::symbol_type yyFlexLexer::yylex()</code> </td><td>no global variables </td></tr>
</table>
<p>Option <code>−−prefix</code> may be used with option <code>−−flex</code> to change the prefix of the generated <code>yyFlexLexer</code> and <code>yylex</code>. This option may be combined with option <code>−−bison</code> to also change the prefix of the generated <code>yytext</code>, <code>yyleng</code>, and <code>yylineno</code>.</p>
<p>Furthermore, <b><code>reflex</code></b> options <code>−−namespace=NAME</code>, <code>−−lexer=LEXER</code> and <code>−−lex=LEX</code> can be used to add a C++ namespace, to rename the lexer class (<code>Lexer</code> or <code>yyFlexLexer</code> by default) and to rename the lexer function (<code>lex</code> or <code>yylex</code> by default), respectively.</p>
<p>For option <code>−−bison-complete</code> the lexer function return type is the parser's <code>symbol_type</code> as defined in the Bison grammar specification. The parser class is specified with option <code>−−bison-cc-parser=PARSER</code> and an optional namespace may be specified with <code>−−bison-cc-namespace=NAME</code>. The lexer function return type may also be explicitly specified with option <code>−−token-type=TYPE</code>.</p>
<p>The following sections explain the <code>−−bison-cc</code>, <code>−−bison-complete</code>, <code>−−bison-bridge</code>, <code>−−bison-locations</code>, and <code>−−reentrant</code> options for <b><code>reflex</code></b>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-bison-cc"></a>
Bison-cc</h3>
<p>The <b><code>reflex</code></b> option <code>−−bison-cc</code> expects a Bison 3.0 <em><code>%skeleton "lalr1.cc"</code></em> C++ parser that is declared as follows in a Bison grammar file:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.yxx file) for C++ */</span></div><div class="line"></div><div class="line">%require  <span class="stringliteral">&quot;3.0&quot;</span></div><div class="line">%skeleton <span class="stringliteral">&quot;lalr1.cc&quot;</span></div><div class="line"></div><div class="line">%code requires{</div><div class="line">  <span class="keyword">namespace </span>yy {</div><div class="line">    <span class="keyword">class </span>Lexer;  <span class="comment">// Generated by reflex with namespace=yy lexer=Lexer lex=yylex</span></div><div class="line">  }</div><div class="line">}</div><div class="line"></div><div class="line">%defines</div><div class="line"></div><div class="line">%parse-param { yy::Lexer&amp; lexer }  <span class="comment">// Construct parser object with lexer</span></div><div class="line"></div><div class="line">%code{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span>  <span class="comment">// header file generated with reflex --header-file</span></div><div class="line"><span class="preprocessor">  #undef yylex</span></div><div class="line"><span class="preprocessor">  #define yylex lexer.yylex  // Within bison&#39;s parse() we should invoke lexer.yylex(), not the global yylex()</span></div><div class="line">}</div><div class="line"></div><div class="line">%<span class="keyword">union </span>{      <span class="comment">// yy::parser::semantic_type yylval is a union:</span></div><div class="line">  <span class="keywordtype">int</span> num;    <span class="comment">// type of yylval.num is int</span></div><div class="line">  <span class="keywordtype">char</span>* str;  <span class="comment">// type of yylval.str is char*</span></div><div class="line">}</div><div class="line"></div><div class="line">%token &lt;num&gt; CONST_NUMBER  <span class="comment">// This defines yy::parser::token::CONST_NUMBER</span></div><div class="line">%token &lt;str&gt; CONST_STRING  <span class="comment">// This defines yy::parser::token::CONST_STRING</span></div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div><div class="line"></div><div class="line"><span class="keywordtype">void</span> yy::parser::error(<span class="keyword">const</span> std::string&amp; msg)</div><div class="line">{</div><div class="line">  std::cerr &lt;&lt; msg &lt;&lt; std::endl;</div><div class="line">}</div></div><!-- fragment --> </div><p>With the <code>−−bison-cc</code> option of <b><code>reflex</code></b>, the <code>yylex()</code> function takes a <code>yy::parser::semantic_type yylval</code> argument that makes the <code>yylval</code> visible in the lexer rules to assign semantic values to.</p>
<p>The scanner is generated with <b><code>reflex</code></b> options <code>−−bison-cc</code>, <code>−−namespace=yy</code> and <code>−−lexer=Lexer</code>. The lexer specification should <code>#include</code> the Bison-generated header file to ensure that the <code>yy::parser::token</code> enums <code>CONST_NUMBER</code> and <code>CONST_STRING</code> are defined.</p>
<p>Using the code above, we can now initialize a Bison parser. We first should create a scanner and pass it to the <code>parser</code> constructor as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">yy::Lexer lexer(std::cin);  <span class="comment">// read from stdin (or a stream, string or FILE)</span></div><div class="line">yy::parser parser(lexer);</div><div class="line"><span class="keywordflow">if</span> (parser.parse() != 0)</div><div class="line">  ... <span class="comment">// error</span></div></div><!-- fragment --> </div><p>We use options <code>−−bison-cc-namespace=NAME</code> and <code>−−bison-cc-parser=NAME</code> to specify the namespace and parser class name of the Bison 3.0 <em><code>%skeleton "lalr1.cc"</code></em> C++ parser you are generating with Bison. These are <code>yy</code> and <code>parser</code> by default, respectively. For option <code>−−bison-cc-namespace=NAME</code> the <code>NAME</code> can be a list of nested namespaces of the form <code>NAME1::NAME2::NAME3</code> or by separating the names by a dot as in <code>NAME1.NAME2.NAME3</code>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-bison-cc-locations"></a>
Bison-cc &amp; locations</h3>
<p>The <b><code>reflex</code></b> option <code>−−bison-cc</code> with <code>−−bison-locations</code> expects a Bison 3.0 <em><code>%skeleton "lalr1.cc"</code></em> C++ parser that is declared as follows in a Bison grammar file:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.yxx file) for C++ */</span></div><div class="line"></div><div class="line">%require  <span class="stringliteral">&quot;3.0&quot;</span></div><div class="line">%skeleton <span class="stringliteral">&quot;lalr1.cc&quot;</span></div><div class="line"></div><div class="line">%code requires{</div><div class="line">  <span class="keyword">namespace </span>yy {</div><div class="line">    <span class="keyword">class </span>Lexer;  <span class="comment">// Generated by reflex with namespace=yy lexer=Lexer</span></div><div class="line">  }</div><div class="line">}</div><div class="line"></div><div class="line">%defines</div><div class="line"></div><div class="line">%locations</div><div class="line"></div><div class="line">%parse-param { yy::Lexer&amp; lexer }  <span class="comment">// Construct parser object with lexer</span></div><div class="line"></div><div class="line">%code{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span>  <span class="comment">// header file generated with reflex --header-file</span></div><div class="line"><span class="preprocessor">  #undef yylex</span></div><div class="line"><span class="preprocessor">  #define yylex lexer.yylex  // Within bison&#39;s parse() we should invoke lexer.yylex(), not the global yylex()</span></div><div class="line">}</div><div class="line"></div><div class="line">%<span class="keyword">union </span>{      <span class="comment">// yy::parser::semantic_type yylval is a union:</span></div><div class="line">  <span class="keywordtype">int</span> num;    <span class="comment">// type of yylval.num is int</span></div><div class="line">  <span class="keywordtype">char</span>* str;  <span class="comment">// type of yylval.str is char*</span></div><div class="line">}</div><div class="line"></div><div class="line">%token &lt;num&gt; CONST_NUMBER  <span class="comment">// This defines yy::parser::token::CONST_NUMBER</span></div><div class="line">%token &lt;str&gt; CONST_STRING  <span class="comment">// This defines yy::parser::token::CONST_STRING</span></div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div><div class="line"></div><div class="line"><span class="keywordtype">void</span> yy::parser::error(<span class="keyword">const</span> location_type&amp; loc, <span class="keyword">const</span> std::string&amp; msg)</div><div class="line">{</div><div class="line">  std::cerr &lt;&lt; msg &lt;&lt; <span class="stringliteral">&quot; at &quot;</span> &lt;&lt; loc &lt;&lt; std::endl;</div><div class="line">}</div></div><!-- fragment --> </div><p>With the <code>−−bison-cc</code> and <code>−−bison-locations</code> options of <b><code>reflex</code></b>, the <code>yylex()</code> function takes <code>yy::parser::semantic_type yylval</code> as the first argument that makes the <code>yylval</code> visible in the lexer rules to assign semantic values to. The second argument <code>yy::location yylloc</code> is set automatically by by invoking the lexer's <code>yylloc_update()</code> in <code>yylex()</code> to update the line and column of the match. The auto-generated virtual <code>yylloc_update()</code> method can be overriden by a user-defined lexer class that extends <code>Lexer</code> (or extends <code>yyFlexLexer</code> when option <code>−−flex</code> is specified).</p>
<p>The scanner is generated with <b><code>reflex</code></b> options <code>−−bison-cc</code>, <code>−−bison-locations</code>, <code>−−namespace=yy</code> and <code>−−lexer=Lexer</code>. The lexer specification should <code>#include</code> the Bison-generated header file to ensure that the <code>yy::parser::token</code> enums <code>CONST_NUMBER</code> and <code>CONST_STRING</code> are defined.</p>
<p>Using the code above, we can now initialize a Bison parser. We first should create a scanner and pass it to the <code>parser</code> constructor as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">yy::Lexer lexer(std::cin);  <span class="comment">// read from stdin (or a stream, string or FILE)</span></div><div class="line">yy::parser parser(lexer);</div><div class="line"><span class="keywordflow">if</span> (parser.parse() != 0)</div><div class="line">  ... <span class="comment">// error</span></div></div><!-- fragment --> </div><p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-bison-complete"></a>
Bison-complete</h3>
<p>The <b><code>reflex</code></b> option <code>−−bison-complete</code> expects a Bison 3.2 C++ parser which uses both <em><code>%define api.value.type variant</code></em> and <em><code>%define api.token.constructor</code></em>. This parser defines the type <code>symbol_type</code> variant and the parser expects <code>yylex</code> to have the type <code>yy::parser::symbol_type yylex()</code>. Here is an example Bison 3.2 C++ complete symbols grammar file:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.yxx file) for C++ */</span></div><div class="line"></div><div class="line">%require <span class="stringliteral">&quot;3.2&quot;</span></div><div class="line">%language <span class="stringliteral">&quot;c++&quot;</span></div><div class="line"></div><div class="line">%define api.namespace {yy}</div><div class="line">%define api.parser.class {parser}</div><div class="line">%define api.value.type variant</div><div class="line">%define api.token.constructor</div><div class="line"></div><div class="line">%defines</div><div class="line">%output <span class="stringliteral">&quot;parser.cpp&quot;</span></div><div class="line"></div><div class="line">%code requires{</div><div class="line">  <span class="keyword">namespace </span>yy {</div><div class="line">    <span class="keyword">class </span>Lexer;  <span class="comment">// Generated by reflex with namespace=yy lexer=Lexer lex=yylex</span></div><div class="line">  }</div><div class="line">}</div><div class="line"></div><div class="line">%parse-param { yy::Lexer&amp; lexer }  <span class="comment">// Construct parser object with lexer</span></div><div class="line"></div><div class="line">%code{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span>  <span class="comment">// header file generated with reflex --header-file</span></div><div class="line"><span class="preprocessor">  #undef yylex</span></div><div class="line"><span class="preprocessor">  #define yylex lexer.yylex  // Within bison&#39;s parse() we should invoke lexer.yylex(), not the global yylex()</span></div><div class="line">}</div><div class="line"></div><div class="line">%define api.token.prefix {TOK_}</div><div class="line">%token &lt;std::string&gt; IDENTIFIER <span class="stringliteral">&quot;identifier&quot;</span>  <span class="comment">// This defines TOK_IDENTIFIER</span></div><div class="line">%token &lt;int&gt; NUMBER <span class="stringliteral">&quot;number&quot;</span>                  <span class="comment">// This defines TOK_NUMBER</span></div><div class="line">%token EOF 0 <span class="stringliteral">&quot;end of file&quot;</span>                    <span class="comment">// This defines TOK_EOF with value 0</span></div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div><div class="line"></div><div class="line"><span class="keywordtype">void</span> yy::parser::error(<span class="keyword">const</span> std::string&amp; msg)</div><div class="line">{</div><div class="line">  std::cerr &lt;&lt; msg &lt;&lt; std::endl;</div><div class="line">  <span class="keywordflow">if</span> (lexer.size() == 0)      <span class="comment">// if token is unknown (no match)</span></div><div class="line">    lexer.matcher().winput(); <span class="comment">// skip character</span></div><div class="line">}</div></div><!-- fragment --> </div><p>With the <code>−−bison-complete</code> option of <b><code>reflex</code></b>, the <code>yylex()</code> function takes no arguments by default and returns a value of type <code>yy::parser::symbol_type</code>. This means that the lexer's action should return values of this type, constructed with <code>yy::parser::symbol_type</code> or with <code>make_TOKENNAME</code> as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">#include &quot;parser.hpp&quot;</span>  <span class="comment">/* Generated by bison. */</span></div><div class="line">}</div><div class="line"></div><div class="line">%option bison-complete</div><div class="line">%option bison-cc-<span class="keyword">namespace</span>=yy</div><div class="line">%option bison-cc-parser=parser</div><div class="line"></div><div class="line">%option exception=<span class="stringliteral">&quot;yy::parser::syntax_error(\&quot;Unknown token.\&quot;)&quot;</span></div><div class="line"></div><div class="line">%option <span class="keyword">namespace</span>=yy</div><div class="line">%option lexer=Lexer</div><div class="line"></div><div class="line">%%</div><div class="line">\s+      <span class="comment">// skip space</span></div><div class="line">[a-z]+   <span class="keywordflow">return</span> yy::parser::make_IDENTIFIER(str());</div><div class="line">[0-9]+   <span class="keywordflow">return</span> yy::parser::make_NUMBER(atoi(text()));</div><div class="line"><span class="stringliteral">&quot;:&quot;</span>      <span class="keywordflow">return</span> yy::parser::symbol_type(<span class="charliteral">&#39;:&#39;</span>);</div><div class="line">&lt;&lt;EOF&gt;&gt;  <span class="keywordflow">return</span> yy::parser::make_EOF();</div><div class="line">%%</div></div><!-- fragment --> </div><p>The scanner is generated with <b><code>reflex</code></b> options <code>−−bison-complete</code>, <code>−−namespace=yy</code> and <code>−−lexer=Lexer</code>. Option <code>−−bison-complete</code> automatically defines the appropriate token type <code>symbol_type</code> depending on <code>−−bison-cc-namespace</code> and on <code>−−bison-cc-parser</code>. We also used options <code>−−bison-cc-namespace=NAME</code> and <code>−−bison-cc-parser=NAME</code> to specify the namespace and parser class name of the Bison 3.2 C++ parser. These are <code>yy</code> and <code>parser</code> by default, respectively (<em><code>%define api.namespace {yy}</code></em> and <em><code>%define api.parser.class {parser}</code></em> are actually superfluous in the example grammer specification because their values are the defaults). We use option <code>−−exception</code> to specify that the scanner's default rule should throw a <code>yy::parser::syntax_error("Unknown token.")</code>. This exception is caught by the parser which calls <code>yy::parser::error</code> with the string <code>"Unknown token."</code> as argument.</p>
<p>We have to be careful with option <code>−−exception</code>. Because no input is consumed, the scanner should not be invoked again or we risk looping on the unmatched input. Alternatively, we can define a "catch all else" rule with pattern <code>.</code> that consumes the offending input:</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line">\s+      <span class="comment">// skip space</span></div><div class="line">[a-z]+   <span class="keywordflow">return</span> yy::parser::make_IDENTIFIER(str());</div><div class="line">[0-9]+   <span class="keywordflow">return</span> yy::parser::make_NUMBER(atoi(text()));</div><div class="line"><span class="stringliteral">&quot;:&quot;</span>      <span class="keywordflow">return</span> yy::parser::symbol_type(<span class="charliteral">&#39;:&#39;</span>);</div><div class="line">&lt;&lt;EOF&gt;&gt;  <span class="keywordflow">return</span> yy::parser::make_EOF();</div><div class="line">.        <span class="keywordflow">throw</span> yy::parser::syntax_error(<span class="stringliteral">&quot;Unknown token.&quot;</span>);</div><div class="line">%%</div></div><!-- fragment --> </div><p>For option <code>−−bison-cc-namespace=NAME</code> the <code>NAME</code> may be a list of nested namespaces of the form <code>NAME1::NAME2::NAME3</code> or by separating the names by a dot as in <code>NAME1.NAME2.NAME3</code>.</p>
<p>Using the code above, we can now initialize a Bison parser in our main program. We first should create a scanner and pass it to the <code>parser</code> constructor as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">yy::Lexer lexer(std::cin);  <span class="comment">// read from stdin (or a stream, string or FILE)</span></div><div class="line">yy::parser parser(lexer);</div><div class="line"><span class="keywordflow">if</span> (parser.parse() != 0)</div><div class="line">  ... <span class="comment">// error</span></div></div><!-- fragment --> </div><p>Note that when the end of input is reached, the lexer returns <code>yy::parser::make_EOF()</code> upon matching <code>&lt;&lt;EOF&gt;&gt;</code>. This rule is optional. When omitted, the return value is <code>yy::parser::symbol_type(0)</code>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-bison-complete-locations"></a>
Bison-complete &amp; locations</h3>
<p>The <b><code>reflex</code></b> option <code>−−bison-complete</code> expects a Bison 3.2 C++ parser which uses both <em><code>%define api.value.type variant</code></em> and <em><code>%define api.token.constructor</code></em>. This parser defines the type <code>symbol_type</code> variant and the parser expects <code>yylex</code> to have the type <code>parser::symbol_type::yylex()</code>. Here is an example Bison 3.2 C++ complete symbols grammar file with Bison <em><code>%locations</code></em> enabled:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.yxx file) for C++ */</span></div><div class="line"></div><div class="line">%require <span class="stringliteral">&quot;3.2&quot;</span></div><div class="line">%language <span class="stringliteral">&quot;c++&quot;</span></div><div class="line"></div><div class="line">%define api.namespace {yy}</div><div class="line">%define api.parser.class {parser}</div><div class="line">%define api.value.type variant</div><div class="line">%define api.token.constructor</div><div class="line"></div><div class="line">%define parse.error verbose</div><div class="line"></div><div class="line">%defines</div><div class="line">%output <span class="stringliteral">&quot;parser.cpp&quot;</span></div><div class="line"></div><div class="line">%locations</div><div class="line">%define api.location.file <span class="stringliteral">&quot;location.hpp&quot;</span></div><div class="line"></div><div class="line">%code requires{</div><div class="line">  <span class="keyword">namespace </span>yy {</div><div class="line">    <span class="keyword">class </span>Lexer;  <span class="comment">// Generated by reflex with namespace=yy lexer=Lexer lex=yylex</span></div><div class="line">  }</div><div class="line">}</div><div class="line"></div><div class="line">%parse-param { yy::Lexer&amp; lexer }  <span class="comment">// Construct parser object with lexer</span></div><div class="line"></div><div class="line">%code{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span>  <span class="comment">// header file generated with reflex --header-file</span></div><div class="line"><span class="preprocessor">  #undef yylex</span></div><div class="line"><span class="preprocessor">  #define yylex lexer.yylex  // Within bison&#39;s parse() we should invoke lexer.yylex(), not the global yylex()</span></div><div class="line">}</div><div class="line"></div><div class="line">%define api.token.prefix {TOK_}</div><div class="line">%token &lt;std::string&gt; IDENTIFIER <span class="stringliteral">&quot;identifier&quot;</span>  <span class="comment">// This defines TOK_IDENTIFIER</span></div><div class="line">%token &lt;int&gt; NUMBER <span class="stringliteral">&quot;number&quot;</span>                  <span class="comment">// This defines TOK_NUMBER</span></div><div class="line">%token EOF 0 <span class="stringliteral">&quot;end of file&quot;</span>                    <span class="comment">// This defines TOK_EOF with value 0</span></div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div><div class="line"></div><div class="line"><span class="keywordtype">void</span> yy::parser::error(<span class="keyword">const</span> location&amp; loc, <span class="keyword">const</span> std::string&amp; msg)</div><div class="line">{</div><div class="line">  std::cerr &lt;&lt; loc &lt;&lt; <span class="stringliteral">&quot;: &quot;</span> &lt;&lt; msg &lt;&lt; std::endl;</div><div class="line">  <span class="keywordflow">if</span> (lexer.size() == 0)      <span class="comment">// if token is unknown (no match)</span></div><div class="line">    lexer.matcher().winput(); <span class="comment">// skip character</span></div><div class="line">}</div></div><!-- fragment --> </div><p>With the <code>−−bison-complete</code> option of <b><code>reflex</code></b>, the <code>yylex()</code> function takes no arguments by default and returns a value of type <code>yy::parser::symbol_type</code>. This means that the lexer's action should return values of this type, constructed with <code>yy::parser::symbol_type</code> or with <code>make_TOKENNAME</code> as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">#include &quot;parser.hpp&quot;</span>    <span class="comment">/* Generated by bison. */</span></div><div class="line"><span class="preprocessor">#include &quot;location.hpp&quot;</span>  <span class="comment">/* Generated by bison %locations. */</span></div><div class="line">}</div><div class="line"></div><div class="line">%option bison-complete</div><div class="line">%option bison-cc-<span class="keyword">namespace</span>=yy</div><div class="line">%option bison-cc-parser=parser</div><div class="line">%option bison-locations</div><div class="line"></div><div class="line">%option exception=<span class="stringliteral">&quot;yy::parser::syntax_error(location(), \&quot;Unknown token.\&quot;)&quot;</span></div><div class="line"></div><div class="line">%option <span class="keyword">namespace</span>=yy</div><div class="line">%option lexer=Lexer</div><div class="line"></div><div class="line">%%</div><div class="line">\s+      <span class="comment">// skip space</span></div><div class="line">[a-z]+   <span class="keywordflow">return</span> yy::parser::make_IDENTIFIER(str(), location());</div><div class="line">[0-9]+   <span class="keywordflow">return</span> yy::parser::make_NUMBER(atoi(text()), location());</div><div class="line"><span class="stringliteral">&quot;:&quot;</span>      <span class="keywordflow">return</span> yy::parser::symbol_type(<span class="charliteral">&#39;:&#39;</span>, location());</div><div class="line">&lt;&lt;EOF&gt;&gt;  <span class="keywordflow">return</span> yy::parser::make_EOF(location());</div><div class="line">%%</div></div><!-- fragment --> </div><p>The scanner is generated with <b><code>reflex</code></b> options <code>−−bison-complete</code>, <code>−−bison-locations</code>, <code>−−namespace=yy</code> and <code>−−lexer=Lexer</code>. Option <code>−−bison-complete</code> automatically defines the appropriate token type <code>symbol_type</code> depending on <code>−−bison-cc-namespace</code> and on <code>−−bison-cc-parser</code>. We also used options <code>−−bison-cc-namespace=NAME</code> and <code>−−bison-cc-parser=NAME</code> to specify the namespace and parser class name of the Bison 3.2 C++ parser. These are <code>yy</code> and <code>parser</code> by default, respectively (i.e. <code>define api.namespace {yy}</code> and <code>define api.parser.class {parser}</code> are actually superfluous in the example grammer specification because their values are the defaults). We use option <code>−−exception</code> to specify that the scanner's default rule should throw a <code>yy::parser::syntax_error(location(), "Unknown token.")</code>. This exception is caught by the parser which calls <code>yy::parser::error</code> with the value of <code>location()</code> and the string <code>"Unknown token."</code> as arguments. The auto-generated virtual lexer class method <code>location()</code> method may be overriden by a user-defined lexer class that extends <code>Lexer</code> (or extends <code>yyFlexLexer</code> when option <code>−−flex</code> is specified).</p>
<p>We have to be careful with option <code>−−exception</code>. Because no input is consumed, the scanner should not be invoked again or we risk looping on the unmatched input. Alternatively, we can define a "catch all else" rule with pattern <code>.</code> that consumes the offending input:</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line">\s+      <span class="comment">// skip space</span></div><div class="line">[a-z]+   <span class="keywordflow">return</span> yy::parser::make_IDENTIFIER(str(), location());</div><div class="line">[0-9]+   <span class="keywordflow">return</span> yy::parser::make_NUMBER(atoi(text()), location());</div><div class="line"><span class="stringliteral">&quot;:&quot;</span>      <span class="keywordflow">return</span> yy::parser::symbol_type(<span class="charliteral">&#39;:&#39;</span>, location());</div><div class="line">&lt;&lt;EOF&gt;&gt;  <span class="keywordflow">return</span> yy::parser::make_EOF(location());</div><div class="line">.        <span class="keywordflow">throw</span> yy::parser::syntax_error(location(), <span class="stringliteral">&quot;Unknown token.&quot;</span>);</div><div class="line">%%</div></div><!-- fragment --> </div><p>For option <code>−−bison-cc-namespace=NAME</code> the <code>NAME</code> may be a list of nested namespaces of the form <code>NAME1::NAME2::NAME3</code> or by separating the names by a dot as in <code>NAME1.NAME2.NAME3</code>.</p>
<p>Using the code above, we can now initialize a Bison parser in our main program. We first should create a scanner and pass it to the <code>parser</code> constructor as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">yy::Lexer lexer(std::cin);  <span class="comment">// read from stdin (or a stream, string or FILE)</span></div><div class="line">yy::parser parser(lexer);</div><div class="line"><span class="keywordflow">if</span> (parser.parse() != 0)</div><div class="line">  ... <span class="comment">// error</span></div></div><!-- fragment --> </div><p>Note that when the end of input is reached, the lexer returns <code>yy::parser::make_EOF()</code> upon matching <code>&lt;&lt;EOF&gt;&gt;</code>. This rule is optional. When omitted, the return value is <code>yy::parser::symbol_type(0, location())</code>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-bison-bridge"></a>
Bison-bridge</h3>
<p>The <b><code>reflex</code></b> option <code>−−bison-bridge</code> expects a Bison "pure parser" that is declared as follows in a Bison grammar file:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.y file) for C or C++ */</span></div><div class="line"></div><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span></div><div class="line"><span class="preprocessor">  #define YYPARSE_PARAM scanner</span></div><div class="line"><span class="preprocessor">  #define YYLEX_PARAM   scanner</span></div><div class="line">%}</div><div class="line"></div><div class="line">%pure-parser</div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div></div><!-- fragment --> </div><p>With the <code>−−bison-bridge</code> option of <b><code>reflex</code></b>, the <code>yyscan_t</code> argument type of <code>yylex()</code> is a <code>void*</code> type that passes the scanner object to this global function (as defined by <code>YYPARSE_PARAM</code> and <code>YYLEX_PARAM</code>). The function then invokes this scanner's lex function. This option also passes the <code>yylval</code> value to the lex function, which is a reference to an <code>YYSTYPE</code> value.</p>
<p>Wtih the <code>−−bison-bridge</code> option two additional functions are generated that should be used to create a new scanner and delete the scanner in your program:</p>
<div class="alt"> <div class="fragment"><div class="line">yyscan_t scanner = <span class="keyword">nullptr</span>;</div><div class="line">yylex_init(&amp;scanner);                <span class="comment">// create a new scanner</span></div><div class="line">...</div><div class="line">int token = yylex(&amp;yylval, scanner); <span class="comment">// scan with bison-bridge</span></div><div class="line">...</div><div class="line">yylex_destroy(scanner);              <span class="comment">// delete a scanner</span></div><div class="line">scanner = <span class="keyword">nullptr</span>;</div></div><!-- fragment --> </div><p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-bison-locations"></a>
Bison-locations</h3>
<p>The option <code>−−bison-locations</code> expects a Bison parser with the locations feature enabled. This feature provides line and column numbers of the matched text for error reporting. For example:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.y file) for C or C++ */</span></div><div class="line"></div><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span></div><div class="line">  <span class="keywordtype">void</span> yyerror(<span class="keyword">const</span> <span class="keywordtype">char</span>*);</div><div class="line">%}</div><div class="line"></div><div class="line">%locations</div><div class="line"></div><div class="line">%<span class="keyword">union </span>{         <span class="comment">// YYSTYPE yylval is a union:</span></div><div class="line">  <span class="keywordtype">int</span> num;       <span class="comment">// type of yylval.num is int</span></div><div class="line">  <span class="keywordtype">char</span>* str;     <span class="comment">// type of yylval.str is char*</span></div><div class="line">}</div><div class="line"></div><div class="line">%{</div><div class="line">  <span class="comment">/* reflex option −−bison-locations makes yylex() take yylval and yylloc */</span></div><div class="line">  <span class="keyword">extern</span> <span class="keywordtype">int</span> yylex(YYSTYPE*, YYLTYPE*);</div><div class="line"><span class="preprocessor">  #define YYLEX_PARAM &amp;yylval, &amp;yylloc</span></div><div class="line">%}</div><div class="line"></div><div class="line"><span class="comment">/* add &amp;yylval and &amp;yyloc parameters to yylex() with a trick: use YYLEX_PARAM */</span></div><div class="line">%lex-param { <span class="keywordtype">void</span> *YYLEX_PARAM }</div><div class="line"></div><div class="line">%token &lt;num&gt; CONST_NUMBER</div><div class="line">%token &lt;str&gt; CONST_STRING</div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div><div class="line"></div><div class="line"><span class="keywordtype">void</span> yyerror(<span class="keyword">const</span> <span class="keywordtype">char</span> *msg)</div><div class="line">{</div><div class="line">  fprintf(stderr, <span class="stringliteral">&quot;%s at %d,%d to line %d,%d\n&quot;</span>,</div><div class="line">      msg,</div><div class="line">      yylloc.first_line,</div><div class="line">      yylloc.first_column,</div><div class="line">      yylloc.last_line,</div><div class="line">      yylloc.last_column);</div><div class="line">}</div></div><!-- fragment --> </div><p>The <code>yylval</code> value is passed to the lex function. The <code>yylloc</code> structure is automatically updated by the RE/flex scanner, so you do not need to define a <code>YY_USER_ACTION</code> macro as you have to with Flex. Instead, this is done automatically in <code>yylex()</code> by invoking the lexer's <code>yylloc_update()</code> to update the line and column of the match. The auto-generated virtual <code>yylloc_update()</code> method may be overriden by a user-defined lexer class that extends <code>Lexer</code> (or extends <code>yyFlexLexer</code> when option <code>−−flex</code> is specified).</p>
<p>Note that with the <code>−−bison-location</code> option, <code>yylex()</code> takes an additional <code>YYLTYPE</code> argument that a Bison parser provides. You can set <code>YYLTYPE</code> as an option <em><code>%option YYLTYPE=type</code></em> in a lexer specification.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-bison-bridge-locations"></a>
Bison-bridge &amp; locations</h3>
<p>Here is a final example that combines options <code>−−bison-locations</code> and <code>−−bison-bridge</code>, The Bison parser should be a Bison pure-parser with locations enabled:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">/* yacc grammar (.y file) for C or C++ */</span></div><div class="line"></div><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span></div><div class="line">  <span class="keywordtype">void</span> yyerror(YYLTYPE*, yyscan_t, <span class="keyword">const</span> <span class="keywordtype">char</span>*);</div><div class="line"><span class="preprocessor">  #define YYPARSE_PARAM scanner</span></div><div class="line"><span class="preprocessor">  #define YYLEX_PARAM   scanner</span></div><div class="line">%}</div><div class="line"></div><div class="line">%locations</div><div class="line">%pure-parser</div><div class="line">%lex-param { <span class="keywordtype">void</span> *scanner }</div><div class="line">%parse-param { <span class="keywordtype">void</span> *scanner }</div><div class="line"></div><div class="line">%<span class="keyword">union </span>{         <span class="comment">// YYSTYPE yylval is a union:</span></div><div class="line">  <span class="keywordtype">int</span> num;       <span class="comment">// yylval.num</span></div><div class="line">  <span class="keywordtype">char</span>* str;     <span class="comment">// yylval.str</span></div><div class="line">}</div><div class="line"></div><div class="line">%token &lt;num&gt; CONST_NUMBER</div><div class="line">%token &lt;str&gt; CONST_STRING</div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// grammar rules</span></div><div class="line">%%</div><div class="line"></div><div class="line"><span class="keywordtype">void</span> yyerror(YYLTYPE *yylloc, yyscan_t scanner, <span class="keyword">const</span> <span class="keywordtype">char</span> *msg)</div><div class="line">{</div><div class="line">  fprintf(stderr, <span class="stringliteral">&quot;%s at %d,%d to line %d,%d\n&quot;</span>,</div><div class="line">    msg,</div><div class="line">    yylloc-&gt;first_line,</div><div class="line">    yylloc-&gt;first_column,</div><div class="line">    yylloc-&gt;last_line,</div><div class="line">    yylloc-&gt;last_column);</div><div class="line">}</div></div><!-- fragment --> </div><dl class="section note"><dt>Note</dt><dd>When Bison <em><code>%locations</code></em> with <em><code>%define api.pure full</code></em> is used, <code>yyerror</code> has the signature <code>void yyerror(YYLTYPE *locp, char const *msg)</code>. This function signature is required to obtain the location information with Bison pure-parsers.</dd>
<dd>
Argument <code>yylval</code> is not a pointer but is passed by reference and should be used as such in the scanner's rules.</dd>
<dd>
Because <code>YYSTYPE</code> is declared by the parser, do not forget to add a <code>#include "y.tab.h"</code> to the top of the specification of your lexer:</dd></dl>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &quot;y.tab.h&quot;</span>    <span class="comment">/* include y.tab.h generated by bison */</span></div><div class="line">}</div></div><!-- fragment --> </div><p>With the <code>−−bison-bridge</code> and <code>−−bison-location</code> options two additional functions are generated that should be used to create a new scanner and delete the scanner in your program:</p>
<div class="alt"> <div class="fragment"><div class="line">yyscan_t scanner = <span class="keyword">nullptr</span>;</div><div class="line">yylex_init(&amp;scanner);      <span class="comment">// create a new scanner</span></div><div class="line">...</div><div class="line">int token = yylex(&amp;yylval, &amp;yylloc, scanner); <span class="comment">// scan with bison-bridge and bison-locations</span></div><div class="line">...</div><div class="line">yylex_destroy(scanner);    <span class="comment">// delete a scanner</span></div><div class="line">scanner = <span class="keyword">nullptr</span>;</div></div><!-- fragment --> </div><p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="reflex-reentrant"></a>
Reentrant scanners</h3>
<p>Option <code>-R</code> or <code>−−reentrant</code> may be used to generate a reentrant scanner that is compatible with reentrant Flex and Bison. This is mainly useful when you combine <code>−−reentrant</code> with <code>−−flex</code> and <code>−−bison</code>. See also <a class="el" href="index.html#reflex-bison">Interfacing with Bison/Yacc </a>.</p>
<p>When using Bison with reentrant scanners, your code should create a <code>yyscan_t</code> scanner object with <code>yylex_init(&amp;scanner)</code> and destroy it with <code>yylex_destroy(scanner)</code>. Reentrant Flex functions take the scanner object as an extra last argument, for example <code>yylex(scanner)</code>:</p>
<div class="alt"> <div class="fragment"><div class="line">yyscan_t scanner = <span class="keyword">nullptr</span>;</div><div class="line">yylex_init(&amp;scanner);       <span class="comment">// create a new scanner</span></div><div class="line">...</div><div class="line">int token = yylex(scanner); <span class="comment">// reentrant scan</span></div><div class="line">...</div><div class="line">yylex_destroy(scanner);     <span class="comment">// delete a scanner</span></div><div class="line">scanner = <span class="keyword">nullptr</span>;</div></div><!-- fragment --> </div><p>Within a rules section we refer to the scanner with macro <code>yyscanner</code>, for example:</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line">{integer}    yylval.i = atoi(yyget_text(yyscanner));</div><div class="line">             <span class="keywordflow">return</span> INTEGER;</div><div class="line">{decimal}    yylval.d = atof(yyget_text(yyscanner));</div><div class="line">             <span class="keywordflow">return</span> DECIMAL;</div><div class="line">.            printf(<span class="stringliteral">&quot;unknown char at line %d\n&quot;</span>, yyget_lineno(yyscanner));</div><div class="line">%%</div></div><!-- fragment --> </div><p>The following functions are available in a reentrant Flex scanner generated with options <code>−−flex</code> and <code>−−reentrant</code>. These functions take an extra argument <code>yyscan_t s</code> that is either <code>yyscanner</code> when the function is used in a rule or in the scope of a lexer method, or is a pointer to the lexer object when the function is used outside the scope of a lexer method:</p>
<table class="doxtable">
<tr>
<th>Reentrant Flex action </th><th>Result  </th></tr>
<tr>
<td><code>yyget_text(s)</code> </td><td>0-terminated text match </td></tr>
<tr>
<td><code>yyget_leng(s)</code> </td><td>size of the match in bytes </td></tr>
<tr>
<td><code>yyget_lineno(s)</code> </td><td>line number of match (&gt;=1) </td></tr>
<tr>
<td><code>yyget_in(s)</code> </td><td>get <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object </td></tr>
<tr>
<td><code>yyset_in(i, s)</code> </td><td>set <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object </td></tr>
<tr>
<td><code>yyget_out(s)</code> </td><td>get <code>std::ostream</code> object </td></tr>
<tr>
<td><code>yyset_out(o, s)</code> </td><td>set output to <code>std::ostream o</code> </td></tr>
<tr>
<td><code>yyget_debug(s)</code> </td><td>reflex option <code>-d</code> sets <code>n=1</code> </td></tr>
<tr>
<td><code>yyset_debug(n, s)</code> </td><td>reflex option <code>-d</code> sets <code>n=1</code> </td></tr>
<tr>
<td><code>yyget_extra(s)</code> </td><td>get user-defined extra parameter </td></tr>
<tr>
<td><code>yyset_extra(x, s)</code> </td><td>set user-defined extra parameter </td></tr>
<tr>
<td><code>yyget_current_buffer(s)</code> </td><td>the current matcher </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#add6f1ba7912f4e259567ebdc120bff17" title="Flex-compatible macro: restart from the given input source. ">yyrestart(i, s)</a></code> </td><td>set input to <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a89cb0e26f2aed856887fdbbc8d10c73a" title="Flex-compatible macro: read one 8-bit character, returns zero when EOF. ">yyinput(s)</a></code> </td><td>get next 8-bit char from input </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a978e5155bc5a5e66b2cadcb7b2124c72" title="Flex-compatible macro: put one 8-bit character back onto the input stream to be read again...">yyunput(c, s)</a></code> </td><td>put back 8-bit char <code>c</code> </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a01e073defdea723b59e8e4415648ac47" title="Flex-compatible macro: output one 8-bit character. ">yyoutput(c, s)</a></code> </td><td>output char <code>c</code> </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#aa28e58c21d4fbe7132b6de8edf76e43a" title="Flex-compatible macro: create and return a new buffer. ">yy_create_buffer(i, n, s)</a></code> </td><td>new matcher <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a66cce193d3578e9236f33137cff97229" title="Flex-compatible macro: delete a buffer. ">yy_delete_buffer(m, s)</a></code> </td><td>delete matcher <code>m</code> </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a3807e0a231906457045e429d6721f291" title="Flex-compatible macro: push the current buffer on the stack to use the given buffer. ">yypush_buffer_state(m, s)</a></code> </td><td>push current matcher, use <code>m</code> </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#ad0d14a86a4c2feaa7e9e73fd26239dc1" title="Flex-compatible macro: pop buffer from the stack and delete the current buffer. ">yypop_buffer_state(s)</a></code> </td><td>pop matcher and delete current </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a3d2545b35aec7c35e6939755777e2a58" title="Flex-compatible macro: scan a string. ">yy_scan_string(s)</a></code> </td><td>scan string <code>s</code> </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#acaf921bfaed6dc25af86fae7ceadc946">yy_scan_wstring(s)</a></code> </td><td>scan wide string <code>s</code> </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a8cef4ba504e54ab2cf998d34ffe047cf">yy_scan_bytes(b, n)</a></code> </td><td>scan <code>n</code> bytes at <code>b</code> (buffered) </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a0872a059892f77ba25bd253e1ac32d2d">yy_scan_buffer(b, n)</a></code> </td><td>scan <code>n</code>-1 bytes at <code>b</code> (zero copy) </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a568a847aae0d31ca8358c9c52d22f15d" title="Flex-compatible macro: push the current start condition state on the stack and transition to the give...">yy_push_state(n, s)</a></code> </td><td>push current state, go to state <code>n</code> </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a650b5dd502b5fdf13563de668b7f104f" title="Flex-compatible macro: pop the stack start condition state and transition to that state...">yy_pop_state(s)</a></code> </td><td>pop state and make it current </td></tr>
<tr>
<td><code><a class="el" href="flexlexer_8h.html#a7d315e7bf2129b110727d0deac8a2031" title="Flex-compatible macro: returns the stack top start condition state. ">yy_top_state(s)</a></code> </td><td>get top state start condition </td></tr>
</table>
<p>With respect to the <code>yyget_extra</code> functions, a scanner object has a <code>YY_EXTRA_TYPE yyextra</code> value that is user-definable. You can define the type in a lexer specification with the <code>extra-type</code> option:</p>
<div class="alt"> <div class="fragment"><div class="line">%option flex bison reentrant</div><div class="line">%option extra-type <span class="keyword">struct </span>extra</div><div class="line">struct extra { ... }; <span class="comment">// type of the data to include in a FlexLexer</span></div></div><!-- fragment --> </div><p>This mechanism is somewhat crude as it originates with Flex' C legacy to add extra user-defined values to a scanner class. Because <b><code>reflex</code></b> is C++, it is recommended to define a derived class that extends the <code>Lexer</code> or <code>FlexLexer</code> class, see <a class="el" href="index.html#reflex-inherit">Inheriting Lexer/yyFlexLexer </a>.</p>
<p>Because scanners are C++ classes, the <code>yyscanner</code> macro is essentially the same is the <code>this</code> pointer. Outside the scope of lexer methods a pointer to your <code>yyFlexLexer lexer</code> object should be used instead.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-search"></a>
Searching versus scanning                                      </h2>
<p>RE/flex generates an efficient search engine with option <code>-S</code> (or <code>−−find</code>). The generated search engine finds all matches while ignoring unmatched input silently, which is different from scanning that matches all input.</p>
<p>Searching with this option is more efficient than scanning with a "catch all
else" dot-rule to ignore unmatched input. For example:</p>
<div class="alt"> <div class="fragment"><div class="line">.    <span class="comment">// no action, ignore unmatched input</span></div></div><!-- fragment --> </div><p>The problem with this rule is that it is invoked for every single unmatched character on the input, which is inefficient and slows down searching for matching patterns significantly when more than a few unmatched characters are encountered in the input. Note that we cannot use <code>.+</code> to match longer patterns because this overlaps with other patterns and is also likely longer than the other patterns, i.e. the rule subsumes those patterns.</p>
<p>Unless the input contains relatively few unmatched characters or bytes to ignore, option <code>-S</code> (or <code>−−find</code>) speeds up searching and matching significantly. This option applies the following optimizations to the RE/flex FSM matcher:</p>
<ul>
<li>Hashing is used to match multiple strings, which is faster than multi-string matching with Aho-Corasick, Commentz-Walter, Wu-Manber, and other.</li>
<li>Single short strings are searched with <code>memchr()</code>. Single long strings are searched with Boyer-Moore-Horspool. Also regex patterns with common prefixes are searched efficiently, e.g. the regex <code>reflex|regex|regular</code> has common prefix string <code>"re"</code> that is searched in the input first, then hashing is used to predict a match for the part after <code>"re"</code>, followed by regex matching with the FSM.</li>
</ul>
<p>With option <code>-S</code> (or <code>−−find</code>), a "catch all else" dot-rule should not be defined, since unmatched input is already ignored with this option and defining a "catch all else" dot-rule actually slows down the search.</p>
<dl class="section note"><dt>Note</dt><dd>By contrast to option <code>-S</code> (or <code>−−find</code>), option <code>-s</code> (or <code>−−nodefault</code>) cannot be used to ignore unmatched input. Option <code>-s</code> produces runtime errors and exceptions for unmatched input.</dd></dl>
<p>This option only applies to the RE/flex matcher and can be combined with options <code>-f</code> (or <code>−−full</code>) and <code>-F</code> (or <code>−−fast</code>) to further increase performance.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-posix-perl"></a>
POSIX versus Perl matching                                 </h2>
<p>The <b><code>reflex</code></b> scanner generator gives you a choice of matchers to use in the generated scanner, where the default is the POSIX RE/flex matcher engine. Other options are the Boost.Regex matcher in POSIX mode or in Perl mode.</p>
<p>To use a matcher for the generated scanner, use one of these three choices:</p>
<table class="doxtable">
<tr>
<th>Option </th><th>Matcher class used </th><th>Mode </th><th>Engine  </th></tr>
<tr>
<td><code>-m reflex</code> </td><td><code>Matcher</code> </td><td>POSIX </td><td>RE/flex lib (default choice) </td></tr>
<tr>
<td><code>-m boost</code> </td><td><code>BoostPosixMatcher</code> </td><td>POSIX </td><td>Boost.Regex </td></tr>
<tr>
<td><code>-m boost-perl</code> </td><td><code>BoostPerlMatcher</code> </td><td>Perl </td><td>Boost.Regex </td></tr>
<tr>
<td><code>-m pcre2-perl</code> </td><td><code>PCRE2Matcher</code> </td><td>Perl </td><td>PCRE2 </td></tr>
</table>
<p>The POSIX matchers look for the <em>longest possible match</em> among the given set of alternative patterns. Perl matchers look for the <em>first match</em> among the given set of alternative patterns.</p>
<p>POSIX is generally preferred for scanners, since it is easier to arrange rules that may have partially overlapping patterns. Since we are looking for the longest match anyway, it does not matter which rule comes first. The order does not matter as long as the length of the matches differ. When matches are of the same length because multiple patterns match, then the first rule is selected.</p>
<p>Consider for example the following <code>lexer.l</code> specification if a lexer with rules that are intended to match keywords and identifiers in some input text:</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line"></div><div class="line"><span class="keywordtype">int</span>                     out() &lt;&lt; <span class="stringliteral">&quot;=&gt; int keyword\n;</span></div><div class="line"><span class="stringliteral">interface               out() &lt;&lt; &quot;</span>=&gt; <span class="keyword">interface </span>keyword\n;</div><div class="line"><span class="keywordtype">float</span>                   out() &lt;&lt; <span class="stringliteral">&quot;=&gt; float keyword\n;</span></div><div class="line"><span class="stringliteral">[A-Za-z][A-Za-z0-9]*    out() &lt;&lt; &quot;</span>=&gt; identifier\n<span class="stringliteral">&quot;;</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">%%</span></div></div><!-- fragment --> </div><p>When the input to the scanner is the text <code>integer</code>, a POSIX matcher selects the last rule that matches it by <em>leftmost longest matching</em> policy. This matching policy selects the rule that matches the longest text. If more than one pattern matches the same length of text then the first pattern that matches takes precedence. This is what we want because it is an identifier in our example programming language: </p><pre class="fragment">reflex -m reflex −−main lexer.l
c++ -o lexer lex.yy.cpp -lreflex
echo "integer" | ./lexer
=&gt; identifier
</pre><p>By contrast, a Perl matcher uses a <em>greedy matching</em> policy, which selects the first rule that matches. In this case it matches the first part <code>int</code> of the text <code>integer</code> and leaves <code>erface</code> to be matched next as an identifier: </p><pre class="fragment">reflex -m boost-perl −−main lexer.l
c++ -o lexer lex.yy.cpp -lreflex -lboost_regex
echo "integer" | ./lexer
=&gt; int keyword
=&gt; identifier
</pre><p>Note that the same greedy matching happens when the text <code>interface</code> is encountered on the input, which we want to recognize as a separate keyword and not match against <code>int</code>: </p><pre class="fragment">reflex -m boost-perl −−main lexer.l
c++ -o lexer lex.yy.cpp -lreflex -lboost_regex
echo "interface" | ./lexer
=&gt; int keyword
=&gt; identifier
</pre><p>Switching the rules for <code>int</code> and <code>interface</code> fixes that specific problem.</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line"></div><div class="line"><span class="keyword">interface               </span>out() &lt;&lt; &quot;=&gt; <span class="keyword">interface </span>keyword\n;</div><div class="line"><span class="keywordtype">int</span>                     out() &lt;&lt; <span class="stringliteral">&quot;=&gt; int keyword\n;</span></div><div class="line"><span class="stringliteral">float                   out() &lt;&lt; &quot;</span>=&gt; <span class="keywordtype">float</span> keyword\n;</div><div class="line">[A-Za-z][A-Za-z0-9]*    out() &lt;&lt; <span class="stringliteral">&quot;=&gt; identifier\n&quot;</span>;</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div> <pre class="fragment">reflex -m boost-perl −−main lexer.l
c++ -o lexer lex.yy.cpp -lreflex -lboost_regex
echo "interface" | ./lexer
=&gt; interface keyword
</pre><p>But we cannot do the same to fix matching <code>integer</code> as an identifier: when moving the last rule up to the top we cannot match <code>int</code> and <code>interface</code> any longer!</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line"></div><div class="line">[A-Za-z][A-Za-z0-9]*    out() &lt;&lt; <span class="stringliteral">&quot;=&gt; identifier\n&quot;</span>;</div><div class="line"><span class="keyword">interface               </span>out() &lt;&lt; &quot;=&gt; <span class="keyword">interface </span>keyword\n;</div><div class="line"><span class="keywordtype">int</span>                     out() &lt;&lt; <span class="stringliteral">&quot;=&gt; int keyword\n;</span></div><div class="line"><span class="stringliteral">float                   out() &lt;&lt; &quot;</span>=&gt; <span class="keywordtype">float</span> keyword\n;</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div> <pre class="fragment">reflex -m boost-perl −−main lexer.l
c++ -o lexer lex.yy.cpp -lreflex -lboost_regex
echo "int" | ./lexer
=&gt; identifier
echo "interface" | ./lexer
=&gt; identifier
</pre><p>Basically, a Perl matcher works in an <em>operational</em> mode by working the regex pattern as a sequence of <em>operations</em> for matching, usually using backtracking to find a matching pattern.</p>
<p>Perl matchers generally support lazy quantifiers and group captures, while most POSIX matchers do not (e.g. Boost.Regex in POSIX mode does not support lazy quantifiers). The RE/flex POSIX matcher supports lazy quantifiers, but not group captures. The added support for lazy quantifiers and word boundary anchors in RE/flex matching offers a reasonably new and useful feature for scanners that require POSIX mode matching.</p>
<p>To prevent a Perl matcher from matching a keyword when an identifier starts with the name of that keyword, we could use a lookahead pattern such as <code>int(?=[^A-Za-z0-9_])</code> which is written in a lexer specification with a trailing context <code>int/[^A-Za-z0-9_]</code> with the <code>/</code> lookahead meta symbol.</p>
<p>A POSIX matcher on the other hand is <em>declarative</em> with a deeper foundation in formal language theory. An advantage of POSIX matchers is that a regular expression can always be compiled to a deterministic finite state machine for efficient matching.</p>
<p>POSIX matching still requires the <code>int</code> matching rule before the identifier matching rule, as in the original lexer specification shown in this section. Otherwise an <code>int</code> on the input will be matched by the identifier rule.</p>
<p>Lookaheads can also be used with POSIX matchers to prioratize rules. Adding a lookahead lengthens the pattern while keeping only the part that matches before the lookahead. For example, the following lexer specification attempts to remove leading <code>0</code> from numbers:</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line"></div><div class="line">0                       <span class="comment">// no action</span></div><div class="line">[0-9]+                  out() &lt;&lt; text() &lt;&lt; std::endl;</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>However, in POSIX mode the first rule only matches if the text is exactly one <code>0</code> because the second rule matches longer texts. The trick here is to use a trailing context with the first rule as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line"></div><div class="line">0/[0-9]+                <span class="comment">// no action</span></div><div class="line">[0-9]+                  out() &lt;&lt; text() &lt;&lt; std::endl;</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div> <pre class="fragment">reflex -m reflex −−main lexer.l
c++ -o lexer lex.yy.cpp -lreflex
echo "00123" | ./lexer
=&gt; 123
echo "0" | ./lexer
=&gt; 0
</pre><p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-debug"></a>
Debugging and profiling                                         </h2>
<p>There are several <b><code>reflex</code></b> options to debug a lexer and analyze its performance given some input text to scan:</p>
<ul>
<li>Option <code>-d</code> (or <code>−−debug</code>) generates a scanner that prints the matched text, which allows you to debug your patterns.</li>
<li>Option <code>-p</code> (or <code>−−perf-report</code>) generates a scanner that profiles the performance of your lexer and the lexer rules executed, which allows you to find hotspots and performance bottlenecks in your rules.</li>
<li>Option <code>-s</code> (or <code>−−nodefault</code>) suppresses the default rule that echoes all unmatched text when no rule matches. The scanner reports "scanner jammed" when no rule matches. Without the <code>−−flex</code> option, a <code>std::runtime</code> exception is thrown.</li>
<li>Option <code>-v</code> (or <code>−−verbose</code>) displays a summary of scanner statistics.</li>
</ul>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Debugging</h3>
<p>Option <code>-d</code> generates a scnner that prints the matched text while scanning input. The output displayed is of the form: </p><pre class="fragment">−−accepting rule at line NNN ("TEXT")
</pre><p>where NNN is the line number of the pattern in the lexer specification and TEXT is the matched text.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Profiling</h3>
<p>Option <code>-p</code> generates a scanner that profiles the performance of your lexer. The performance report shows the performance statistics obtained for each pattern defined in the lexer specification, i.e. the number of matches per pattern, the total text length of the matches per pattern, and the total time spent matching and executing the rule corresponding to the pattern. The performance profile statistics are collected when the scanner runs on some given input. Profiling allows you to effectively fine-tune the performance of your lexer by focussing on patterns and rules that are frequently matched that turn out to be computationally expensive.</p>
<p>This is perhaps best illustrated with an example. The JSON parser <em><code>json.l</code></em> located in the examples directory of the RE/flex download package was built with reflex option <code>-p</code> and then run on some given JSON input to analyze its performance: </p><pre class="fragment">reflex 0.9.22 json.l performance report:
  INITIAL rules matched:
    rule at line 51 accepted 58 times matching 255 bytes total in 0.009 ms
    rule at line 52 accepted 58 times matching 58 bytes total in 0.824 ms
    rule at line 53 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 54 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 55 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 56 accepted 5 times matching 23 bytes total in 0.007 ms
    rule at line 57 accepted 38 times matching 38 bytes total in 0.006 ms
    rule at line 72 accepted 0 times matching 0 bytes total in 0 ms
    default rule accepted 0 times
  STRING rules matched:
    rule at line 60 accepted 38 times matching 38 bytes total in 0.021 ms
    rule at line 61 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 62 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 63 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 64 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 65 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 66 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 67 accepted 0 times matching 0 bytes total in 0 ms
    rule at line 68 accepted 314 times matching 314 bytes total in 0.04 ms
    rule at line 69 accepted 8 times matching 25 bytes total in 0.006 ms
    rule at line 72 accepted 0 times matching 0 bytes total in 0 ms
    default rule accepted 0 times
  WARNING: execution times are relative:
    1) includes caller's execution time between matches when yylex() returns
    2) perf-report instrumentation adds overhead and increases execution times
</pre><p>The timings shown include the time of the pattern match and the time of the code executed by the rule. If the rule returns to the caller than the time spent by the caller is also included in this timing. For this example, we have two start condition states namely INITIAL and STRING. The rule at line 52 is:</p>
<div class="alt"> <div class="fragment"><div class="line">[][}{,:]        { <span class="keywordflow">return</span> <a class="code" href="flexlexer_8h.html#a0d71f919dbec1ffd74b2460fa7e5ac28">yytext</a>[0]; }</div></div><!-- fragment --> </div><p>This returns a [ or ] bracket, a { or } brace, a comma, or a colon to the parser. Since the pattern and rule are very simple, we do not expect these to contribute much to the 0.824 ms time spent on this rule. The parser code executed when the scanner returns could be expensive. Depending on the character returned, the parser constructs a JSON array (bracket) or a JSON object (brace), and populates arrays and objects with an item each time a comma is matched. But which is most expensive? To obtain timings of these events separately, we can split the rule up into three similar rules:</p>
<div class="alt"> <div class="fragment"><div class="line">[][]        { <span class="keywordflow">return</span> <a class="code" href="flexlexer_8h.html#a0d71f919dbec1ffd74b2460fa7e5ac28">yytext</a>[0]; }</div><div class="line">[}{]        { <span class="keywordflow">return</span> <a class="code" href="flexlexer_8h.html#a0d71f919dbec1ffd74b2460fa7e5ac28">yytext</a>[0]; }</div><div class="line">[,:]        { <span class="keywordflow">return</span> <a class="code" href="flexlexer_8h.html#a0d71f919dbec1ffd74b2460fa7e5ac28">yytext</a>[0]; }</div></div><!-- fragment --> </div><p>Then we use reflex option <code>-p</code>, recompile the generated scanner <em><code>lex.yy.cpp</code></em> and rerun our experiment to see these changes: </p><pre class="fragment">    rule at line 52 accepted 2 times matching 2 bytes total in 0.001 ms
    rule at line 53 accepted 14 times matching 14 bytes total in 0.798 ms
    rule at line 54 accepted 42 times matching 42 bytes total in 0.011 ms
</pre><p>So it turns out that the construction of a JSON object by the parser is relatively speaking the most expensive part of our application, when { and } are encountered on the input. We should focus our optimization effort there if we want to improve the overall speed of our JSON parser.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-examples"></a>
Examples                                                     </h2>
<p>Some lexer specification examples to generate scanners with RE/flex.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 1</h3>
<p>The following Flex specification counts the lines, words, and characters on the input. We use <code>yyleng</code> match text length to count 8-bit characters (bytes).</p>
<p>To build this example with RE/flex, use <b><code>reflex</code></b> option <code>−−flex</code> to generate Flex-compatible "yy" variables and functions. This generates a C++ scanner class <code>yyFlexLexer</code> that is compatible with the Flex scanner class (assuming Flex with option <code>-+</code> for C++).</p>
<div class="alt"> <div class="fragment"><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &lt;stdio.h&gt;</span></div><div class="line">  <span class="keywordtype">int</span> ch = 0, wd = 0, nl = 0;</div><div class="line">%}</div><div class="line"></div><div class="line">%option noyywrap</div><div class="line">%option <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a></div><div class="line"></div><div class="line">nl      \r?\n</div><div class="line">wd      [^ \t\r\n]+</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">{nl}    ch += <a class="code" href="flexlexer_8h.html#afa07a629486cb790560bb95713ec7794">yyleng</a>; ++nl;</div><div class="line">{wd}    ch += <a class="code" href="flexlexer_8h.html#afa07a629486cb790560bb95713ec7794">yyleng</a>; ++wd;</div><div class="line">.       ++ch;</div><div class="line">&lt;&lt;EOF&gt;&gt; printf(<span class="stringliteral">&quot;%8d%8d%8d\n&quot;</span>, nl, wd, ch); <a class="code" href="flexlexer_8h.html#add62fdebca5f1fac074d8c918dc45eec">yyterminate</a>();</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>To generate a scanner with a global <code>yylex()</code> function similar to Flex in C mode (i.e. without Flex option <code>-+</code>), use <b><code>reflex</code></b> option <code>−−bison</code> with the specification shown above. This option when combined with <code>−−flex</code> produces the global "yy" functions and variables. This means that you can use RE/flex scanners with Bison (Yacc) and with any other C code, assuming everything is compiled together with a C++ compiler.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 2</h3>
<p>An improved implementation drops the use of global variables in favor of Lexer class member variables. We also want to count Unicode letters with the <code>wd</code> counter instead of ASCII letters, which are single bytes while Unicode UTF-8 encodings vary in size. So we add the Unicode option and use <code>\w</code> to match Unicode word characters. Note that <code>.</code> (dot) matches Unicode, so the match length may be longer than one character that must be counted. We drop the <code>−−flex</code> option and use RE/flex Lexer methods instead of the Flex "yy" functions:</p>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &lt;iostream&gt;</span></div><div class="line"><span class="preprocessor">  #include &lt;iomanip&gt;</span></div><div class="line">  <span class="keyword">using namespace </span><a class="code" href="namespacestd.html">std</a>;</div><div class="line">}</div><div class="line"></div><div class="line">%<span class="keyword">class</span>{</div><div class="line">  <span class="keywordtype">int</span> ch, wd, nl;</div><div class="line">}</div><div class="line"></div><div class="line">%init{</div><div class="line">  ch = wd = nl = 0;</div><div class="line">}</div><div class="line"></div><div class="line">%option <a class="code" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39">unicode</a></div><div class="line">%option <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a></div><div class="line">%option full</div><div class="line"></div><div class="line">nl      \r?\n</div><div class="line">wd      (\w|\p{Punctuation})+</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">{nl}    ch += size(); ++nl;</div><div class="line">{wd}    ch += size(); ++wd;</div><div class="line">.       ch += size();</div><div class="line">&lt;&lt;EOF&gt;&gt; out() &lt;&lt; setw(8) &lt;&lt; nl &lt;&lt; setw(8) &lt;&lt; wd &lt;&lt; setw(8) &lt;&lt; ch &lt;&lt; endl;</div><div class="line">        <span class="keywordflow">return</span> 0;</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>This simple word count program differs slightly from the Unix wc utility, because the wc utility counts words delimited by wide character spaces (<code>iswspace</code>) whereas this program counts words made up from word characters combined with punctuation.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 3</h3>
<p>The following RE/flex specification filters tags from XML documents and verifies whether or not the tags are properly balanced. Note that this example uses the lazy repetitions to keep the patterns simple. The XML document scanned should not include invalid XML characters such as <code>/</code>, <code>&lt;</code>, or <code>&gt;</code> in attributes (otherwise the tags will not match properly). The <code>dotall</code> option allows <code>.</code> (dot) to match newline in all patterns, similar to the <code>(?s)</code> modifier in regexes.</p>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &lt;stdio.h&gt;</span></div><div class="line">}</div><div class="line"></div><div class="line">%<span class="keyword">class</span>{</div><div class="line">  <span class="keywordtype">int</span> level;</div><div class="line">}</div><div class="line"></div><div class="line">%init{</div><div class="line">  level = 0;</div><div class="line">}</div><div class="line"></div><div class="line">%o matcher=<a class="code" href="namespacereflex.html">reflex</a> <a class="code" href="namespacereflex_1_1convert__flag.html#ab6a63c3a106c2dbcbad7b86797453f58">dotall</a> <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a></div><div class="line">%x ATTRIBUTES</div><div class="line"></div><div class="line">name                    [A-Za-z_:\x80-\xFF][-.0-9A-Za-z_:\x80-\xFF]*</div><div class="line">pi                      &lt;\?{name}</div><div class="line">comment                 </div><div class="line">open                    &lt;{name}</div><div class="line">close                   &lt;\/{name}&gt;</div><div class="line">cdata                   &lt;!\[CDATA\[.*?]]&gt;</div><div class="line"><span class="keywordtype">string</span>                  \<span class="stringliteral">&quot;.*?\&quot;|&#39;.*?&#39;</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">%%</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">{comment}               |</span></div><div class="line"><span class="stringliteral">{cdata}                 // skip comments and CDATA sections</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">{pi}                    level++;</span></div><div class="line"><span class="stringliteral">                        start(ATTRIBUTES);</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">{open}                  printf(&quot;</span>%*s%s\n<span class="stringliteral">&quot;, level++, &quot;</span><span class="stringliteral">&quot;, text() + 1);</span></div><div class="line"><span class="stringliteral">                        start(ATTRIBUTES);</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">{close}                 matcher().less(size() - 1);</span></div><div class="line"><span class="stringliteral">                        printf(&quot;</span>%*s%s\n<span class="stringliteral">&quot;, --level, &quot;</span><span class="stringliteral">&quot;, text() + 2);</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">&lt;&lt;EOF&gt;&gt;                 printf(&quot;</span>Tags are %sbalanced\n<span class="stringliteral">&quot;, level ? &quot;</span>im<span class="stringliteral">&quot; : &quot;</span><span class="stringliteral">&quot;);</span></div><div class="line"><span class="stringliteral">                        return 0;</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">&lt;ATTRIBUTES&gt;&quot;</span>/&gt;<span class="stringliteral">&quot;        --level;</span></div><div class="line"><span class="stringliteral">                        start(INITIAL);</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">&lt;ATTRIBUTES&gt;&quot;</span>&gt;<span class="stringliteral">&quot;         start(INITIAL);</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">&lt;ATTRIBUTES&gt;{name}      |</span></div><div class="line"><span class="stringliteral">&lt;ATTRIBUTES&gt;{string}    // skip attribute names and strings</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">&lt;*&gt;.                    // skip anything else</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">%%</span></div></div><!-- fragment --> </div><p>Note thay we restrict XML tag names to valid characters, including all UTF-8 sequences that run in the range <code>\x80</code>-<code>\xFF</code> per 8-bit character. This matches all Unicode characters U+0080 to U+10FFFF.</p>
<p>The <code>ATTRIBUTES</code> state is used to scan attributes and their quoted values separately from the <code>INITIAL</code> state. The <code>INITIAL</code> state permits quotes to freely occur in character data, whereas the <code>ATTRIBUTES</code> state matches quoted attribute values.</p>
<p>We use <code>matcher().less(size() - 1)</code> to remove the ending <code>&gt;</code> from the match in <code>text()</code>. The <code>&gt;</code> will be matched again, this time by the <code>&lt;*&gt;.</code> rule that ignores it. We could also have used a lookahead pattern <code>"\&lt;/"{name}/"\&gt;"</code> where <code>X/Y</code> means look ahead for <code>Y</code> after <code>X</code>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 4</h3>
<p>This example Flex specification scans non-Unicode C/C++ source code. It uses free space mode to enhance readability.</p>
<div class="alt"> <div class="fragment"><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &lt;stdio.h&gt;</span></div><div class="line">%}</div><div class="line"></div><div class="line">%o flex <a class="code" href="namespacereflex_1_1convert__flag.html#a32e4f47592b2442c814a32502365a8e3">freespace</a> <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a></div><div class="line"></div><div class="line">directive       ^ \h* # (. | \\ \r? \n)+</div><div class="line">name            [\u\l_] \w*</div><div class="line">udec            0 | [1-9] \d*</div><div class="line">uhex            0 [Xx] [[:xdigit:]]+</div><div class="line">uoct            0 [0-7]+</div><div class="line"><span class="keywordtype">int</span>             [-+]? ({udec} | {uhex}) \</div><div class="line">                  ([Ll]{0,2} [Uu]? | [Uu] [Ll]{0,2})</div><div class="line"><span class="keywordtype">float</span>           [-+] \d* (\d | \.\d | \d\.) \d* \</div><div class="line">                  ([Ee][-+]? \d+)? \</div><div class="line">                  [FfLl]?</div><div class="line"><span class="keywordtype">char</span>            L? <span class="stringliteral">&#39; (\\. | [^\\\n&#39;</span>])* <span class="stringliteral">&#39;</span></div><div class="line"><span class="stringliteral">string          L? \&quot; (\\. | \\\r?\n | [^\\\n&quot;])* \&quot;</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">%%</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">\s+</span></div><div class="line"><span class="stringliteral">&quot;//&quot; .*? \n</span></div><div class="line"><span class="stringliteral">&quot;/*&quot; (.|\n)*? &quot;*/&quot;</span></div><div class="line"><span class="stringliteral">{directive}     { printf(&quot;DIRECTIVE %s\n&quot;, yytext); }</span></div><div class="line"><span class="stringliteral">{name}          { printf(&quot;NAME      %s\n&quot;, yytext); }</span></div><div class="line"><span class="stringliteral">{int}           { printf(&quot;INT       %s\n&quot;, yytext); }</span></div><div class="line"><span class="stringliteral">{float}         { printf(&quot;FLOAT     %s\n&quot;, yytext); }</span></div><div class="line"><span class="stringliteral">{char}          { printf(&quot;CHAR      %s\n&quot;, yytext); }</span></div><div class="line"><span class="stringliteral">{string}        { printf(&quot;STRING    %s\n&quot;, yytext); }</span></div><div class="line"><span class="stringliteral">[[:punct:]]     { printf(&quot;PUNCT     %s\n&quot;, yytext); }</span></div><div class="line"><span class="stringliteral">.               { printf(&quot;*** ERROR &#39;</span>%s<span class="stringliteral">&#39; at line %d\n&quot;, yytext, yylineno); }</span></div><div class="line"><span class="stringliteral"></span></div><div class="line"><span class="stringliteral">%%</span></div></div><!-- fragment --> </div><p>Free space mode permits spacing between concatenations and alternations. To match a single space, use <code>" "</code> or <code>[ ]</code>. Long patterns can continue on the next line(s) when lines ends with an escape <code>\</code>.</p>
<p>In free space mode you MUST place actions in <em><code>{ }</code></em> blocks and user code in <em><code>%{ %}</code></em> blocks instead of indented.</p>
<p>When used with option <code>unicode</code>, the scanner automatically recognizes and scans Unicode identifier names. Note that we can use <code>matcher().columno()</code> or <code>matcher().border()</code> in the error message to indicate the location on a line of the match. The <code>matcher().columno()</code> method takes tab spacing and wide characters into account. To obtain the byte offset from the start of the line use <code>matcher().border()</code>. The <code>matcher()</code> object associated with the Lexer offers several other methods that Flex does not support.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 5</h3>
<p>This example defines a search engine to find C/C++ directives, such as <code>#define</code> and <code>#include</code>, in the input fast.</p>
<div class="alt"> <div class="fragment"><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &lt;stdio.h&gt;</span></div><div class="line">%}</div><div class="line"></div><div class="line">%o fast find <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a></div><div class="line"></div><div class="line">directive       ^\h*#(.|\\\r?\n)+</div><div class="line"></div><div class="line">%%</div><div class="line"></div><div class="line">{directive}     echo();</div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>Option <code>%o find</code> (<code>-S</code> or <code>−−find</code>) specifies that unmatched input text should be ignored silently instead of being echoed to standard output, see <a class="el" href="index.html#reflex-search">Searching versus scanning </a>. Option <code>%fast</code> (<code>-F</code> or <code>−−fast</code>) generates an efficient FSM in direct code.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="reflex-limitations"></a>
Limitations                                               </h2>
<p>The RE/flex matcher engine uses an efficient FSM. There are known limitations to FSM matching that apply to Flex/Lex and therefore also apply to the <b><code>reflex</code></b> scanner generator and to the RE/flex matcher engine:</p>
<ul>
<li>Lookaheads (trailing contexts) must appear at the end of a pattern when using the RE/flex matcher, so <code>a/b</code> (which is the same as lookahead <code>a(?=b)</code>) is permitted, but <code>(a/b)c</code> and <code>a(?=b)c</code> are not.</li>
<li>Lookaheads cannot be properly matched when the ending of the first part of the pattern matches the beginning of the second part, such as <code>zx*/xy*</code>, where the <code>x*</code> matches the <code>x</code> at the beginning of the lookahead pattern.</li>
<li>Anchors and boundaries must appear at the start or at the end of a pattern. The begin of buffer/line anchors <code>\A</code> and <code>^</code>, end of buffer/line anchors <code>\z</code> and <code>$</code> and the word boundary anchors must start or end a pattern. For example, <code>\&lt;cow\&gt;</code> is permitted, but <code>.*\Bboy</code> is not.</li>
<li>The POSIX Lex <code>REJECT</code> action is not supported.</li>
<li>The POSIX Lex table size parameters <code>p</code>, <code>n</code>, <code>a</code>, <code>e</code>, <code>k</code>, and <code>o</code> are not supported; <code>o</code> may be used as a shorthand for <code>option</code>.</li>
<li>Flex translations <em><code>%T</code></em> are not supported.</li>
</ul>
<p>Some of these limitations may be removed in future versions of RE/flex.</p>
<p>Boost.Regex library limitations:</p>
<ul>
<li>Lookbehinds <code>φ(?&lt;=ψ)</code> with the Boost.Regex matcher engines should not look too far behind. Any input before the current line may be discarded and is no longer available when new input is shifted into the internal buffer. Only input on the current line from the start of the line to the match is guaranteed.</li>
<li>Boost.Regex partial matching may fail to find the longest match when greedy repetition patterns such as <code>.*</code> are used. Under certain conditions greedy repetitions may behave as lazy repetitions. For example, the Boost.Regex engine may return the short match <code>abc</code> when the regex <code>a.*c</code> is applied to <code>abc abc</code>, instead of returning the full match <code>abc abc</code>. The problem is caused by the limitations of Boost.Regex <code>match_partial</code> matching algorithm. To work around this limitation, we suggest to make the repetition pattern as specific as possible and not overlap with the pattern that follows the repetition. <em>The easiest solution is to read the entire input</em> using <b><code>reflex</code></b> option <code>-B</code> (batch input). For a stand-alone <code>BoostMatcher</code>, use the <code>buffer()</code> method. We consider this Boost.Regex partial match behavior a bug, not a restriction, because <em>as long as backtracking on a repetition pattern is possible given some partial text, Boost.Regex should flag the result as a partial match instead of a full match.</em></li>
</ul>
<p>PCRE2 library limitations:</p>
<ul>
<li>Lookbehinds <code>φ(?&lt;=ψ)</code> with the PCRE2 matcher engines should not look too far behind. Any input before the current line may be discarded and is no longer available when new input is shifted into the internal buffer. Only input on the current line from the start of the line to the match is guaranteed.</li>
</ul>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h1><a class="anchor" id="regex"></a>
The RE/flex regex library                                              </h1>
<p>The RE/flex regex library consists of a set of C++ templates and classes that encapsulate regex engines in a standard API for scanning, tokenizing, searching, and splitting of strings, wide strings, files, and streams.</p>
<p>The RE/flex regex library is a class hierarchy that has at the root an abstract class <code><a class="el" href="classreflex_1_1_abstract_matcher.html" title="The abstract matcher base class template defines an interface for all pattern matcher engines...">reflex::AbstractMatcher</a></code>. Pattern types may differ between for matchers so the <code><a class="el" href="classreflex_1_1_pattern_matcher.html" title="The pattern matcher class template extends abstract matcher base class. ">reflex::PatternMatcher</a></code> template class takes a pattern type and creates a class that is complete except for the implementation of the <code>reflex::match()</code> virtual method that requires a regex engine, such as Boost.Regex, PCRE2, or the RE/flex engine.</p>
<p>To compile your application, simply include the applicable regex matcher of your choice in your source code as we will explain in the next sections. To compile, link your application against the <code>libreflex</code> library: </p><pre class="fragment">c++ myapp.cpp -lreflex
</pre><p>And optionally <code>-lboost_regex</code> (or <code>-lboost_regex-mt</code> depending on your Boost installation) if you use Boost.Regex for matching: </p><pre class="fragment">c++ myapp.cpp -lreflex -lboost_regex
</pre><p>or <code>-lpcre2-8</code> if you use PCRE2 for matching: </p><pre class="fragment">c++ myapp.cpp -lreflex -lpcre2-8
</pre><p>If <code>libreflex</code> was not installed then linking with <code>-lreflex</code> fails. See <a class="el" href="index.html#link-errors">Undefined symbols and link errors </a> on how to resolve this.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="regex-boost"></a>
Boost matcher classes                                            </h2>
<p>The <code><a class="el" href="classreflex_1_1_boost_matcher.html" title="Boost matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::BoostMatcher</a></code> inherits <code><a class="el" href="classreflex_1_1_pattern_matcher.html" title="The pattern matcher class template extends abstract matcher base class. ">reflex::PatternMatcher</a>&lt;boost::regex&gt;</code>, and in turn the <code><a class="el" href="classreflex_1_1_boost_perl_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost Perl regex matching. ">reflex::BoostPerlMatcher</a></code> and <code><a class="el" href="classreflex_1_1_boost_posix_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost POSIX regex matching. ">reflex::BoostPosixMatcher</a></code> are both derived from <code><a class="el" href="classreflex_1_1_boost_matcher.html" title="Boost matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::BoostMatcher</a></code>:</p>
<div class="image">
<img src="classreflex_1_1_boost_matcher__inherit__graph.png" />
</div>
<p>An instance of <code><a class="el" href="classreflex_1_1_boost_perl_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost Perl regex matching. ">reflex::BoostPerlMatcher</a></code> is initialized with flag <code>match_perl</code> and the flag <code>match_not_dot_newline</code>, these are <code>boost::regex_constants</code> flags. These flags are the only difference with the plain <code><a class="el" href="classreflex_1_1_boost_matcher.html" title="Boost matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::BoostMatcher</a></code>.</p>
<p>An instance of <code><a class="el" href="classreflex_1_1_boost_posix_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost POSIX regex matching. ">reflex::BoostPosixMatcher</a></code> creates a POSIX matcher. This means that lazy quantifiers are not supported and the <em>leftmost longest rule</em> applies to pattern matching. This instance is initialized with the flags <code>match_posix</code> and <code>match_not_dot_newline</code>.</p>
<p>Boost.Regex is a powerful library. The RE/flex regex API enhances this library with operations to match, search, scan, and split data from a given input. The input may be a file, a string, or a stream. Files that are UTF-8/16/32-encoded are automatically decoded. Further, streams can be potentially of unlimited length because internal buffering is used by the RE/flex regex API enhancements to efficiently apply Boost.Regex partial pattern matching to streaming data. This enhancement permits pattern matching of interactive input from the console, such that searching and scanning interactive input for matches will return these matches immediately.</p>
<dl class="section note"><dt>Note</dt><dd>The <code><a class="el" href="classreflex_1_1_boost_matcher.html" title="Boost matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::BoostMatcher</a></code> extends the capabilities of Boost.Regex, which does not natively support streaming input:</dd>
<dd>
The Boost.Regex library requires the target text to be loaded into memory for pattern matching. It is possible to match a target text incrementally with the <code>match_partial</code> flag and <code>boost::regex_iterator</code>. Incremental matching can be used to support matching on (possibly infinite) streams. To use this method correctly, a buffer should be created that is large enough to hold the text for each match. The buffer must adjust with a growing size of the matched text, to ensure that long matches that do not fit the buffer are not discared.</dd>
<dd>
Boost.Regex <code>Boost.IOStreams</code> with <code>regex_filter</code> loads the entire stream into memory which does not permit pattern matching of streaming and interactive input data.</dd></dl>
<p>A <code><a class="el" href="classreflex_1_1_boost_matcher.html" title="Boost matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::BoostMatcher</a></code> (or <code><a class="el" href="classreflex_1_1_boost_perl_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost Perl regex matching. ">reflex::BoostPerlMatcher</a></code>) engine is created from a <code>boost::regex</code> object, or string regex, and some given input for normal (Perl mode) matching:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher( boost::regex or <span class="keywordtype">string</span>, <a class="code" href="classreflex_1_1_input.html">reflex::Input</a> [, <span class="stringliteral">&quot;options&quot;</span>] )</div></div><!-- fragment --><p>Likewise, a <code><a class="el" href="classreflex_1_1_boost_posix_matcher.html" title="Boost matcher engine class, extends reflex::BoostMatcher for Boost POSIX regex matching. ">reflex::BoostPosixMatcher</a></code> engine is created from a <code>boost::regex</code> object, or string regex, and some given input for POSIX mode matching:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_boost_posix_matcher.html">reflex::BoostPosixMatcher</a> matcher( boost::regex or <span class="keywordtype">string</span>, <a class="code" href="classreflex_1_1_input.html">reflex::Input</a> [, <span class="stringliteral">&quot;options&quot;</span>] )</div></div><!-- fragment --><p>For input you can specify a string, a wide string, a file, or a stream object.</p>
<p>We use option <code>"N"</code> to permit empty matches when searching input with <code><a class="el" href="classreflex_1_1_abstract_matcher.html#a3ec40e4c9070975bcafa94af5db0e514" title="functor to search input ">reflex::BoostMatcher::find</a></code>.</p>
<p>You can convert an expressive regex of the form defined in <a class="el" href="index.html#reflex-patterns">Patterns </a> to a regex that the boost::regex engine can handle:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> std::string regex = <a class="code" href="classreflex_1_1_boost_matcher.html#ad1827615da2d51f4896e9c802b6463fc">reflex::BoostMatcher::convert</a>( <span class="keywordtype">string</span>, [ flags ]);</div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher( regex, <a class="code" href="classreflex_1_1_input.html">reflex::Input</a> [, <span class="stringliteral">&quot;options&quot;</span>] )</div></div><!-- fragment --><p>The converter is specific to the matcher selected, i.e. <code><a class="el" href="classreflex_1_1_boost_matcher.html#ad1827615da2d51f4896e9c802b6463fc" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::BoostMatcher::convert</a></code>, <code><a class="el" href="classreflex_1_1_boost_matcher.html#ad1827615da2d51f4896e9c802b6463fc" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::BoostPerlMatcher::convert</a></code>, and <code><a class="el" href="classreflex_1_1_boost_posix_matcher.html#a51f86b84e5bd4914ef37495c6847eb2a" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::BoostPosixMatcher::convert</a></code>. The converters also translates Unicode <code>\p</code> character classes to UTF-8 patterns, converts bracket character classes containing Unicode, and groups UTF-8 multi-byte sequences in the regex string.</p>
<p>The converter throws a <code><a class="el" href="classreflex_1_1regex__error.html" title="Regex syntax error exceptions. ">reflex::regex_error</a></code> exception if conversion fails, for example when the regex syntax is invalid.</p>
<p>To compile your application, link your application against the <code>libreflex</code> library and <code>-lboost_regex</code> (or <code>-lboost_regex-mt</code> depending on your Boost installation): </p><pre class="fragment">c++ myapp.cpp -lreflex -lboost_regex
</pre><p>See <a class="el" href="index.html#reflex-patterns">Patterns </a> for more details on regex patterns.</p>
<p>See <a class="el" href="index.html#regex-input">The Input class </a> for more details on the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class.</p>
<p>See <a class="el" href="index.html#regex-methods">Methods and iterators </a> for more details on pattern matching methods.</p>
<p>See <a class="el" href="index.html#regex-convert">Regex converters </a> for more details on regex converters.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="regex-pcre2"></a>
PCRE2 matcher classes                                            </h2>
<p>The <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html" title="PCRE2 JIT-optimized matcher engine class implements reflex::PatternMatcher pattern matching interface...">reflex::PCRE2Matcher</a></code> inherits <code><a class="el" href="classreflex_1_1_pattern_matcher.html" title="The pattern matcher class template extends abstract matcher base class. ">reflex::PatternMatcher</a>&lt;std::string&gt;</code>. The <code><a class="el" href="classreflex_1_1_p_c_r_e2_u_t_f_matcher.html" title="PCRE2 JIT-optimized native PCRE2_UTF+PCRE2_UCP matcher engine class, extends PCRE2Matcher. ">reflex::PCRE2UTFMatcher</a></code> is derived from <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html" title="PCRE2 JIT-optimized matcher engine class implements reflex::PatternMatcher pattern matching interface...">reflex::PCRE2Matcher</a></code>:</p>
<div class="image">
<img src="classreflex_1_1_p_c_r_e2_matcher__inherit__graph.png" />
</div>
<p>An instance of <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html" title="PCRE2 JIT-optimized matcher engine class implements reflex::PatternMatcher pattern matching interface...">reflex::PCRE2Matcher</a></code> is initialized with a pattern that is compiled with <code>pcre2_compile()</code> and <code>pcre2_jit_compile()</code> for optimal performance with PCRE2 JIT-generated code.</p>
<p>An instance of <code><a class="el" href="classreflex_1_1_p_c_r_e2_u_t_f_matcher.html" title="PCRE2 JIT-optimized native PCRE2_UTF+PCRE2_UCP matcher engine class, extends PCRE2Matcher. ">reflex::PCRE2UTFMatcher</a></code> creates a PCRE2 matcher with native Unicode support, using PCRE2 options <code>PCRE2_UTF+PCRE2_UCP</code>.</p>
<p>PCRE2 is a powerful library. The RE/flex regex API enhances this library with operations to match, search, scan, and split data from a given input. The input may be a file, a string, or a stream. Files that are UTF-8/16/32-encoded are automatically decoded. Further, streams can be of potentially unlimited length because internal buffering is used by the RE/flex regex API enhancements to efficiently apply PCRE2 partial pattern matching to streaming data. This enhancement permits pattern matching of interactive input from the console, such that searching and scanning interactive input for matches will return these matches immediately.</p>
<p>A <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html" title="PCRE2 JIT-optimized matcher engine class implements reflex::PatternMatcher pattern matching interface...">reflex::PCRE2Matcher</a></code> (or <code><a class="el" href="classreflex_1_1_p_c_r_e2_u_t_f_matcher.html" title="PCRE2 JIT-optimized native PCRE2_UTF+PCRE2_UCP matcher engine class, extends PCRE2Matcher. ">reflex::PCRE2UTFMatcher</a></code>) engine is created from a string regex and some given input:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher( <span class="keywordtype">string</span>, <a class="code" href="classreflex_1_1_input.html">reflex::Input</a> [, <span class="stringliteral">&quot;options&quot;</span>] )</div></div><!-- fragment --><p>Likewise, a <code><a class="el" href="classreflex_1_1_p_c_r_e2_u_t_f_matcher.html" title="PCRE2 JIT-optimized native PCRE2_UTF+PCRE2_UCP matcher engine class, extends PCRE2Matcher. ">reflex::PCRE2UTFMatcher</a></code> engine is created from a string regex and some given input:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_u_t_f_matcher.html">reflex::PCRE2UTFMatcher</a> matcher( <span class="keywordtype">string</span>, <a class="code" href="classreflex_1_1_input.html">reflex::Input</a> [, <span class="stringliteral">&quot;options&quot;</span>] )</div></div><!-- fragment --><p>This matcher uses PCRE2 native Unicode matching. Non-UTF input is not supported, such as plain binary. UTF encoding errors in the input will cause the matcher to terminate.</p>
<p>For input you can specify a string, a wide string, a file, or a stream object.</p>
<p>We use option <code>"N"</code> to permit empty matches when searching input with <code><a class="el" href="classreflex_1_1_abstract_matcher.html#a3ec40e4c9070975bcafa94af5db0e514" title="functor to search input ">reflex::PCRE2Matcher::find</a></code>.</p>
<p>You can convert an expressive regex of the form defined in <a class="el" href="index.html#reflex-patterns">Patterns </a> to a regex that the PCRE2 engine can handle:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> std::string regex = <a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html#a176455169227dc6e9975c80b77d639be">reflex::PCRE2Matcher::convert</a>( <span class="keywordtype">string</span>, [ flags ]);</div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher( regex, <a class="code" href="classreflex_1_1_input.html">reflex::Input</a> [, <span class="stringliteral">&quot;options&quot;</span>] )</div></div><!-- fragment --><p>The converter is specific to the matcher selected, i.e. <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html#a176455169227dc6e9975c80b77d639be" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::PCRE2Matcher::convert</a></code> and <code><a class="el" href="classreflex_1_1_p_c_r_e2_u_t_f_matcher.html#afbf0dbed19598a0b322179c3fcb8037f" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::PCRE2UTFMatcher::convert</a></code>. The former converter converts Unicode <code>\p</code> character classes to UTF-8 patterns, converts bracket character classes containing Unicode, and groups UTF-8 multi-byte sequences in the regex string. The latter converter does not convert these regex constructs, which are matched by the PCRE2 engine initialized with options <code>PCRE2_UTF+PCRE2_UCP</code>.</p>
<p>The converter throws a <code><a class="el" href="classreflex_1_1regex__error.html" title="Regex syntax error exceptions. ">reflex::regex_error</a></code> exception if conversion fails, for example when the regex syntax is invalid.</p>
<p>To compile your application, link your application against the <code>libreflex</code> library and <code>-lpcre2-8</code>: </p><pre class="fragment">c++ myapp.cpp -lreflex -lpcre2-8
</pre><p>See <a class="el" href="index.html#reflex-patterns">Patterns </a> for more details on regex patterns.</p>
<p>See <a class="el" href="index.html#regex-input">The Input class </a> for more details on the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class.</p>
<p>See <a class="el" href="index.html#regex-methods">Methods and iterators </a> for more details on pattern matching methods.</p>
<p>See <a class="el" href="index.html#regex-convert">Regex converters </a> for more details on regex converters.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="regex-std"></a>
std::regex matcher classes                                         </h2>
<p>The <code><a class="el" href="classreflex_1_1_std_matcher.html" title="std matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::StdMatcher</a></code> class inherits <code><a class="el" href="classreflex_1_1_pattern_matcher.html" title="The pattern matcher class template extends abstract matcher base class. ">reflex::PatternMatcher</a>&lt;std::regex&gt;</code> as a base. The <code><a class="el" href="classreflex_1_1_std_ecma_matcher.html" title="std matcher engine class, extends reflex::StdMatcher for ECMA std::regex::ECMAScript syntax and regex...">reflex::StdEcmaMatcher</a></code> and <code><a class="el" href="classreflex_1_1_std_posix_matcher.html" title="std matcher engine class, extends reflex::StdMatcher for POSIX ERE std::regex::awk syntax and regex m...">reflex::StdPosixMatcher</a></code> are derived classes from <code><a class="el" href="classreflex_1_1_std_matcher.html" title="std matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::StdMatcher</a></code>:</p>
<div class="image">
<img src="classreflex_1_1_std_matcher__inherit__graph.png" />
</div>
<p>An instance of <code><a class="el" href="classreflex_1_1_std_ecma_matcher.html" title="std matcher engine class, extends reflex::StdMatcher for ECMA std::regex::ECMAScript syntax and regex...">reflex::StdEcmaMatcher</a></code> is initialized with regex syntax option <code>std::regex::ECMAScript</code>. This is also the default std::regex syntax.</p>
<p>An instance of <code><a class="el" href="classreflex_1_1_std_posix_matcher.html" title="std matcher engine class, extends reflex::StdMatcher for POSIX ERE std::regex::awk syntax and regex m...">reflex::StdPosixMatcher</a></code> creates a POSIX AWK-based matcher. So that lazy quantifiers are not supported and the <em>leftmost longest rule</em> applies to pattern matching. This instance is initialized with the regex syntax option <code>std::regex::awk</code>.</p>
<p>The C++11 std::regex library does not support <code>match_partial</code> that is needed to match patterns on real streams with an adaptive internal buffer that grows when longer matches are made when more input becomes available. Therefore all input is buffered with the C++11 std::regex class matchers.</p>
<p>With respect to performance, as of this time of writing, std::regex matching is much slower than other matchers, slower by a factor 10 or more.</p>
<p>The std::regex syntax is more limited than Boost.Regex, PCRE2, and RE/flex. Also the matching behavior differs and cannot be controlled with mode modifiers:</p>
<ul>
<li><code>.</code> (dot) matches anything except <code>\0</code> (NUL);</li>
<li><code>\177</code> is erroneously interpreted as a backreference, <code>\0177</code> does not match;</li>
<li><code>\x7f</code> is not supported in POSIX mode;</li>
<li><code>\cX</code> is not supported in POSIX mode;</li>
<li><code>\Q..\E</code> is not supported;</li>
<li>no mode modifiers <code>(?imsux-imsux:φ)</code>;</li>
<li>no <code>\A</code>, <code>\z</code>, <code>\&lt;</code> and <code>\&gt;</code> anchors;</li>
<li>no <code>\b</code> and <code>\B</code> anchors in POSIX mode;</li>
<li>no non-capturing groups <code>(?:φ)</code> in POSIX mode;</li>
<li>empty regex patterns and matcher option <code>"N"</code> (nullable) may cause issues (std::regex <code>match_not_null</code> appears not to work as documented);</li>
<li><code>interactive()</code> is not supported.</li>
</ul>
<p>To work around these limitations that the std::regex syntax imposes, you can convert an expressive regex of the form defined in section <a class="el" href="index.html#reflex-patterns">Patterns </a> to a regex that the std::regex engine can handle:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="stdmatcher_8h.html">reflex/stdmatcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> std::string regex = <a class="code" href="classreflex_1_1_std_matcher.html#a5b24b7a11cc3c8b7c2b949fb9cfae60a">reflex::StdMatcher::convert</a>( <span class="keywordtype">string</span>, [ flags ]);</div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_std_matcher.html">reflex::StdMatcher</a> matcher( regex, <a class="code" href="classreflex_1_1_input.html">reflex::Input</a> [, <span class="stringliteral">&quot;options&quot;</span>] )</div></div><!-- fragment --><p>The converter is specific to the matcher selected, i.e. <code><a class="el" href="classreflex_1_1_std_matcher.html#a5b24b7a11cc3c8b7c2b949fb9cfae60a" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::StdMatcher::convert</a></code>, <code><a class="el" href="classreflex_1_1_std_matcher.html#a5b24b7a11cc3c8b7c2b949fb9cfae60a" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::StdEcmaMatcher::convert</a></code>, and <code><a class="el" href="classreflex_1_1_std_posix_matcher.html#af0c8ddd6ae3216b70dc36357e41c71c6" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::StdPosixMatcher::convert</a></code>. The converters also translates Unicode <code>\p</code> character classes to UTF-8 patterns, converts bracket character classes containing Unicode, and groups UTF-8 multi-byte sequences in the regex string.</p>
<p>The converter throws a <code><a class="el" href="classreflex_1_1regex__error.html" title="Regex syntax error exceptions. ">reflex::regex_error</a></code> exception if conversion fails, for example when the regex syntax is invalid.</p>
<p>To compile your application, link your application against the <code>libreflex</code> and enable <code>std::regex</code> with <code>-std=c++11</code>: </p><pre class="fragment">c++ -std=c++11 myapp.cpp -lreflex
</pre><p>See <a class="el" href="index.html#reflex-patterns">Patterns </a> for more details on regex patterns.</p>
<p>See <a class="el" href="index.html#regex-input">The Input class </a> for more details on the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class.</p>
<p>See <a class="el" href="index.html#regex-methods">Methods and iterators </a> for more details on pattern matching methods.</p>
<p>See <a class="el" href="index.html#regex-convert">Regex converters </a> for more details on regex converters.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="regex-matcher"></a>
The reflex::Matcher class                                      </h2>
<p>The RE/flex framework includes a POSIX regex matching library <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> that inherits the API from <code><a class="el" href="classreflex_1_1_pattern_matcher.html" title="The pattern matcher class template extends abstract matcher base class. ">reflex::PatternMatcher</a>&lt;<a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a>&gt;</code>:</p>
<div class="image">
<img src="classreflex_1_1_matcher__inherit__graph.png" />
</div>
<p>where the RE/flex <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> class represents a regex pattern. Patterns as regex texts are internally compiled into deterministic finite state machines by the <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> class. The machines are used by the <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> for fast matching of regex patterns on some given input. The <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> is faster than the Boost.Regex and PCRE2 matchers.</p>
<p>A <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> engine is constructed from a <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> object, or a string regex, and some given input:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a> matcher( <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> or <span class="keywordtype">string</span>, <a class="code" href="classreflex_1_1_input.html">reflex::Input</a> [, <span class="stringliteral">&quot;options&quot;</span>] )</div></div><!-- fragment --><p>The regex is specified as a string or a <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> object, see <a class="el" href="index.html#regex-pattern">The reflex::Pattern class </a> below.</p>
<p>We use option <code>"N"</code> to permit empty matches when searching input with <code><a class="el" href="classreflex_1_1_abstract_matcher.html#a3ec40e4c9070975bcafa94af5db0e514" title="functor to search input ">reflex::Matcher::find</a></code>. Option <code>"T=8"</code> sets the tab size to 8 for <a class="el" href="index.html#reflex-pattern-dents">Indent/nodent/dedent</a> matching.</p>
<p>For input you can specify a string, a wide string, a file, or a stream object.</p>
<p>A regex string with Unicode patterns can be converted for Unicode matching as follows:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> std::string regex = <a class="code" href="classreflex_1_1_matcher.html#a8862686f6d5680d4d0e5af146c4aed69">reflex::Matcher::convert</a>( <span class="keywordtype">string</span>, [ flags ]);</div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a> matcher( regex, <a class="code" href="classreflex_1_1_input.html">reflex::Input</a> [, <span class="stringliteral">&quot;options&quot;</span>] )</div></div><!-- fragment --><p>The converter is specific to the matcher and translates Unicode <code>\p</code> character classes to UTF-8 patterns, converts bracket character classes containing Unicode, and groups UTF-8 multi-byte sequences in the regex string.</p>
<p>To compile your application, link your application against the <code>libreflex</code>: </p><pre class="fragment">c++ myapp.cpp -lreflex
</pre><p>See <a class="el" href="index.html#reflex-patterns">Patterns </a> for more details on regex patterns.</p>
<p>See <a class="el" href="index.html#regex-input">The Input class </a> for more details on the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class.</p>
<p>See <a class="el" href="index.html#regex-methods">Methods and iterators </a> for more details on pattern matching methods.</p>
<p>See <a class="el" href="index.html#regex-convert">Regex converters </a> for more details on regex converters.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="regex-pattern"></a>
The reflex::Pattern class                                      </h2>
<p>The <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> class is used by the <code>reflex::matcher</code> for pattern matching. The <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> class converts a regex pattern to an efficient FSM and takes a regex string and options to construct the FSM internally. The pattern instance is passed to a <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> constructor:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span></div><div class="line"></div><div class="line">[<span class="keyword">static</span>] <a class="code" href="namespacereflex.html">reflex</a>:Pattern pattern(<span class="keywordtype">string</span> [, <span class="stringliteral">&quot;options&quot;</span>] )</div><div class="line"></div><div class="line"><a class="code" href="namespacereflex.html">reflex</a>::Matcher matcher(pattern, <a class="code" href="namespacereflex.html">reflex</a>::Input [, &quot;options&quot;] )</div></div><!-- fragment --><p>It may also be used to replace a matcher's current pattern, see <a class="el" href="index.html#intro2">And a flexible regex library </a>.</p>
<p>To improve performance, it is recommended to create a <code>static</code> instance of the pattern if the regex string is fixed. This avoids repeated FSM construction at run time.</p>
<p>The following options are combined in a string and passed to the <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> constructor:</p>
<table class="doxtable">
<tr>
<th>Option </th><th>Effect  </th></tr>
<tr>
<td><code>b</code> </td><td>bracket lists are parsed without converting escapes </td></tr>
<tr>
<td><code>e=c;</code> </td><td>redefine the escape character </td></tr>
<tr>
<td><code>f=file.cpp;</code> </td><td>save finite state machine code to <code>file.cpp</code> </td></tr>
<tr>
<td><code>f=file.gv;</code> </td><td>save deterministic finite state machine to <code>file.gv</code> </td></tr>
<tr>
<td><code>i</code> </td><td>case-insensitive matching, same as <code>(?i)X</code> </td></tr>
<tr>
<td><code>m</code> </td><td>multiline mode, same as <code>(?m)X</code> </td></tr>
<tr>
<td><code>n=name;</code> </td><td>use <code>reflex_code_name</code> for the machine (instead of FSM) </td></tr>
<tr>
<td><code>q</code> </td><td>Flex/Lex-style quotations "..." equals <code>\Q...\E</code> </td></tr>
<tr>
<td><code>r</code> </td><td>throw regex syntax error exceptions, otherwise ignore errors </td></tr>
<tr>
<td><code>s</code> </td><td>dot matches all (aka. single line mode), same as <code>(?s)X</code> </td></tr>
<tr>
<td><code>x</code> </td><td>inline comments, same as <code>(?x)X</code> </td></tr>
<tr>
<td><code>w</code> </td><td>display regex syntax errors before raising them as exceptions </td></tr>
</table>
<p>The compilation of a <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> object into a FSM may throw an exception with option <code>"r"</code> when the regex string has problems:</p>
<div class="fragment"><div class="line"><span class="keywordflow">try</span></div><div class="line">{</div><div class="line">  <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> pattern(argv[1], <span class="stringliteral">&quot;r&quot;</span>); <span class="comment">// &quot;r&quot; option throws syntax error exceptions</span></div><div class="line">  ...</div><div class="line">  <span class="comment">// code that uses the pattern object</span></div><div class="line">  ...</div><div class="line">}</div><div class="line"><span class="keywordflow">catch</span> (<a class="code" href="classreflex_1_1regex__error.html">reflex::regex_error</a>&amp; e)</div><div class="line">{</div><div class="line">  <span class="keywordflow">switch</span> (e.<a class="code" href="classreflex_1_1regex__error.html#a9db1cdde85bfe6f8f7807357addca122">code</a>())</div><div class="line">  {</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a293014262113a8affe812cdc1a155199">reflex::regex_error::mismatched_parens</a>:     std::cerr &lt;&lt; <span class="stringliteral">&quot;mismatched ( )&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#aa103b656bc8c84276caa3a3e479d5aed">reflex::regex_error::mismatched_braces</a>:     std::cerr &lt;&lt; <span class="stringliteral">&quot;mismatched { }&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a388b62b0f150f2c4e673a7326df98bf0">reflex::regex_error::mismatched_brackets</a>:   std::cerr &lt;&lt; <span class="stringliteral">&quot;mismatched [ ]&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a4da14a39e65667e922d4ac9251d3acf2">reflex::regex_error::mismatched_quotation</a>:  std::cerr &lt;&lt; <span class="stringliteral">&quot;mismatched \\Q...\\E quotation&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#aba10b497ace31b4e829961820f59ac1f">reflex::regex_error::empty_expression</a>:      std::cerr &lt;&lt; <span class="stringliteral">&quot;regex (sub)expression should not be empty&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#ab8d5ad921586a5821a5a6ea57f3ef7f3">reflex::regex_error::empty_class</a>:           std::cerr &lt;&lt; <span class="stringliteral">&quot;character class [...] is empty, e.g. [a&amp;&amp;[b]]&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#ac7c47c5a86f7f50f4128e4b31f3b8cd8">reflex::regex_error::invalid_class</a>:         std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid character class name&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a0a3dbc259753fe9c56bcb86ab22933f6">reflex::regex_error::invalid_class_range</a>:   std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid character class range, e.g. [Z-A]&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a3f67bcffa141ac3fd804dfe28a009165">reflex::regex_error::invalid_escape</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid escape character&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#abe00de8d9374f3b3528c181c3bbcee7c">reflex::regex_error::invalid_anchor</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid anchor or boundary&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a72584e74ccf493595d88f0ccc4b61f55">reflex::regex_error::invalid_repeat</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid repeat, e.g. {10,1}&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a1c01a511c833d13708a5607c7a6d810a">reflex::regex_error::invalid_quantifier</a>:    std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid lazy or possessive quantifier&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#af9454b097595a8c39fe0118cdbae60d0">reflex::regex_error::invalid_modifier</a>:      std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid (?ismux:) modifier&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a216ae66d5ecfe124a192a1e12fed6a00">reflex::regex_error::invalid_collating</a>:     std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid collating element&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a90a346e348a466432d45f9251e3df4e5">reflex::regex_error::invalid_backreference</a>: std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid backreference&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#ac3e3283b3859fae05b21655d8ee096a6">reflex::regex_error::invalid_syntax</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid regex syntax&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a2a490e1b97afa08a8f068077a78dc63c">reflex::regex_error::exceeds_length</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;exceeds length limit, pattern is too long&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a0c63bf6cff5be3ddb509de3ea84d6e41">reflex::regex_error::exceeds_limits</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;exceeds complexity limits, e.g. {n,m} range too large&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">  }</div><div class="line">  std::cerr &lt;&lt; std::endl &lt;&lt; e.what();</div><div class="line">}</div></div><!-- fragment --><p>By default, the <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> constructor solely throws the <code><a class="el" href="classreflex_1_1regex__error.html#a2a490e1b97afa08a8f068077a78dc63c" title="regex exceeds length limit (reflex::Pattern class only) ">reflex::regex_error::exceeds_length</a></code> and <code><a class="el" href="classreflex_1_1regex__error.html#a0c63bf6cff5be3ddb509de3ea84d6e41" title="regex exceeds complexity limits (reflex::Pattern class only) ">reflex::regex_error::exceeds_limits</a></code> exceptions and silently ignores syntax errors.</p>
<p>Likewise, the <code><a class="el" href="classreflex_1_1_matcher.html#a8862686f6d5680d4d0e5af146c4aed69" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::Matcher::convert</a></code>, <code><a class="el" href="classreflex_1_1_boost_matcher.html#ad1827615da2d51f4896e9c802b6463fc" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::BoostPerlMatcher::convert</a></code>, <code><a class="el" href="classreflex_1_1_boost_matcher.html#ad1827615da2d51f4896e9c802b6463fc" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::BoostMatcher::convert</a></code>, <code><a class="el" href="classreflex_1_1_boost_posix_matcher.html#a51f86b84e5bd4914ef37495c6847eb2a" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::BoostPosixMatcher::convert</a></code>, <code><a class="el" href="classreflex_1_1_p_c_r_e2_matcher.html#a176455169227dc6e9975c80b77d639be" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::PCRE2Matcher::convert</a></code>, and <code><a class="el" href="classreflex_1_1_p_c_r_e2_u_t_f_matcher.html#afbf0dbed19598a0b322179c3fcb8037f" title="Convert a regex to an acceptable form, given the specified regex library signature &quot;[decls:]escapes[...">reflex::PCRE2UTFMatcher::convert</a></code> functions may throw a <code>reflex_error</code> exception. See the next section for details.</p>
<p>The <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> class has the following public methods:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>assign(r,o)</code> </td><td>(re)assign regex string <code>r</code> with string of options <code>o</code> </td></tr>
<tr>
<td><code>assign(r)</code> </td><td>(re)assign regex string <code>r</code> with default options </td></tr>
<tr>
<td><code>=r</code> </td><td>same as above </td></tr>
<tr>
<td><code>size()</code> </td><td>returns the number of top-level sub-patterns </td></tr>
<tr>
<td><code>[0]</code> </td><td>operator returns the regex string of the pattern </td></tr>
<tr>
<td><code>[n]</code> </td><td>operator returns the <code>n</code>th sub-pattern regex string </td></tr>
<tr>
<td><code>reachable(n)</code></td><td>true if sub-pattern <code>n</code> is reachable in the FSM </td></tr>
</table>
<p>The assignment methods may throw exceptions, which are the same as the constructor may throw.</p>
<p>The <code><a class="el" href="classreflex_1_1_pattern.html#a9f602f0f0589a2f388742c9db722b685" title="Check if subpattern is reachable by a match. ">reflex::Pattern::reachable</a></code> method verifies which top-level grouped alternations are reachable. This means that the sub-pattern of an alternation has a FSM accepting state that identifies the sub-pattern. For example:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> pattern(<span class="stringliteral">&quot;(a+)|(a)&quot;</span>, <span class="stringliteral">&quot;r&quot;</span>);</div><div class="line">std::cout &lt;&lt; <span class="stringliteral">&quot;regex = &quot;</span> &lt;&lt; pattern[0] &lt;&lt; std::endl;</div><div class="line"><span class="keywordflow">for</span> (<span class="keywordtype">size_t</span> i = 1; i &lt;= pattern.size(); ++i)</div><div class="line">  <span class="keywordflow">if</span> (!pattern.reachable(i))</div><div class="line">    std::cerr &lt;&lt; pattern[i] &lt;&lt; <span class="stringliteral">&quot; is not reachable&quot;</span> &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">regex = (a+)|(a)
(a) is not reachable
</pre><p>For this example regex, <code>(a)</code> is not reachable as the pattern is subsumed by <code>(a+)</code>. The <code><a class="el" href="classreflex_1_1_abstract_matcher.html#af51252ce359e6106b58c81f70776ccfb" title="Returns a positive integer (true) indicating the capture index of the matched text in the pattern or ...">reflex::Matcher::accept</a></code> method will never return 2 when matching the input <code>a</code> and always return 1, as per leftmost longest match policy. The same observation holds for the <code><a class="el" href="classreflex_1_1_abstract_matcher.html#ab880e6c8fe0df7c4ee7bb629aab41d0f" title="Returns nonzero capture index (i.e. true) if the entire input matches this matcher&#39;s pattern (and int...">reflex::Matcher::matches</a></code>, <code><a class="el" href="classreflex_1_1_abstract_matcher.html#a3ec40e4c9070975bcafa94af5db0e514" title="functor to search input ">reflex::Matcher::find</a></code>, <code><a class="el" href="classreflex_1_1_abstract_matcher.html#a489c6327fb6521cc722852ada690f5ca" title="functor to scan input (to tokenize input) ">reflex::Matcher::scan</a></code>, and <code><a class="el" href="classreflex_1_1_abstract_matcher.html#a16d0cb13c2dca93cc556c06c24633b8d" title="functor to split input ">reflex::Matcher::split</a></code> method and functors. Reversing the alternations resolves this: <code>(a)|(a+)</code>.</p>
<dl class="section note"><dt>Note</dt><dd>The <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code> regex forms support capturing groups at the top-level only, i.e. among the top-level alternations.</dd></dl>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="regex-convert"></a>
Regex converters                                               </h2>
<p>To work around limitations of regex libraries and to support Unicode matching, RE/flex offers converters to translate expressive regex syntax forms (with Unicode patterns defined in section <a class="el" href="index.html#reflex-patterns">Patterns </a>) to regex strings that the selected regex engines can handle.</p>
<p>The converters translate <code>\p</code> Unicode classes, translate character class set operations such as <code>[a-z−−[aeiou]]</code>, convert escapes such as <code>\X</code>, and enable/disable <code>(?imsux-imsux:φ)</code> mode modifiers to a regex string that the underlying regex library understands and can use.</p>
<p>Each converter is specific to the regex engine. You can use a converter for the matcher of your choice:</p>
<ul>
<li><code>std::string reflex::BoostMatcher::convert(const std::string&amp; regex, reflex::convert_flag_type flags)</code> converts an enhanced <code>regex</code> for use with Boost.Regex;</li>
<li><code>std::string reflex::BoostPerlMatcher::convert(const std::string&amp; regex, reflex::convert_flag_type flags)</code> converts an enhanced <code>regex</code> for use with Boost.Regex in Perl mode;</li>
<li><code>std::string reflex::BoostPosixMatcher::convert(const std::string&amp; regex, reflex::convert_flag_type flags)</code> converts an enhanced <code>regex</code> for use with Boost.Regex in POSIX mode;</li>
<li><code>std::string reflex::PCRE2Matcher::convert(const std::string&amp; regex, reflex::convert_flag_type flags)</code> converts an enhanced <code>regex</code> for use with PCRE2;</li>
<li><code>std::string reflex::PCRE2UTFMatcher::convert(const std::string&amp; regex, reflex::convert_flag_type flags)</code> converts an enhanced <code>regex</code> for use with PCRE2 native Unicode matching;</li>
<li><code>std::string reflex::StdMatcher::convert(const std::string&amp; regex, reflex::convert_flag_type flags)</code> converts an enhanced <code>regex</code> for use with C++ std::regex;</li>
<li><code>std::string reflex::StdEcmaMatcher::convert(const std::string&amp; regex, reflex::convert_flag_type flags)</code> converts an enhanced <code>regex</code> for use with C++ std::regex in ECMA mode;</li>
<li><code>std::string reflex::StdPosixMatcher::convert(const std::string&amp; regex, reflex::convert_flag_type flags)</code> converts an enhanced <code>regex</code> for use with C++ std::regex in POSIX mode;</li>
<li><code>std::string reflex::Matcher::convert(const std::string&amp; regex, reflex::convert_flag_type flags)</code> converts an enhanced <code>regex</code> for use with the RE/flex POSIX regex library;</li>
</ul>
<p>where <code>flags</code> is optional. When specified, it may be a combination of the following <code><a class="el" href="namespacereflex_1_1convert__flag.html">reflex::convert_flag</a></code> flags:</p>
<table class="doxtable">
<tr>
<th>Flag </th><th>Effect  </th></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#af5c6fe22fc79bb893e135d583d525930" title="no conversion (default) ">reflex::convert_flag::none</a></code> </td><td>no conversion </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#ae9e0ae7e5abd19e8499a638e8dfff5d3" title="convert basic regex (BRE) to extended regex (ERE) ">reflex::convert_flag::basic</a></code> </td><td>convert basic regular expression syntax (BRE) to extended regular expression syntax (ERE) </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39" title="convert . (dot), \s, \w, \l, \u, \S, \W, \L, \U to Unicode ">reflex::convert_flag::unicode</a></code> </td><td><code>.</code>, <code>\s</code>, <code>\w</code>, <code>\l</code>, <code>\u</code>, <code>\S</code>, <code>\W</code>, <code>\L</code>, <code>\U</code> match Unicode, same as <code>(?u)</code> </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#a1f27c083368a4cb0d50b4a613adef897" title="remove capturing groups, add capturing groups to the top level ">reflex::convert_flag::recap</a></code> </td><td>remove capturing groups and add capturing groups to the top level </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#a7222821c0cc6579424c71c261e9b8805" title="convert Lex/Flex regular expression syntax ">reflex::convert_flag::lex</a></code> </td><td>convert Flex/Lex regular expression syntax </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#a1e3e66422498b8e4c6fd7770393ea988" title="convert \uXXXX and UTF-16 surrogate pairs ">reflex::convert_flag::u4</a></code> </td><td>convert <code>\uXXXX</code> (shorthand for <code>\u{XXXX}</code>), may conflict with <code>\u</code> (upper case letter). </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#aaed1f496284028394303d24be3e7bbb9" title="inverted character classes do not match newline \n ">reflex::convert_flag::notnewline</a></code> </td><td>inverted character classes do not match newline <code>\n</code>, e.g. <code>[^a-z]</code> does not match <code>\n</code> </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#a2fcbe121a0a80ba46baa1a4b67b045df" title="convert Unicode to compact UTF-8 patterns and DFA, permits some invalid UTF-8 sequences ...">reflex::convert_flag::permissive</a></code> </td><td>when used with <code>unicode</code>, produces a more compact FSM that tolerates some invalid UTF-8 sequences </td></tr>
</table>
<p>The following <code><a class="el" href="namespacereflex_1_1convert__flag.html">reflex::convert_flag</a></code> flags correspond to the common <code>(?imsx)</code> modifiers. These flags or modifiers may be specified, or both. Modifiers are removed from the converted regex if the regex library does not support them:</p>
<table class="doxtable">
<tr>
<th>Flag </th><th>Effect  </th></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#a70281ca46dbef31569206d4c110c2308" title="convert regex to ignore case, same as (?i) ">reflex::convert_flag::anycase</a></code> </td><td>convert regex to ignore case, same as <code>(?i)</code> </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#a32e4f47592b2442c814a32502365a8e3" title="convert regex by removing spacing, same as (?x) ">reflex::convert_flag::freespace</a></code> </td><td>convert regex by removing all freespace-mode spacing, same as <code>(?x)</code> </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#ab6a63c3a106c2dbcbad7b86797453f58" title="convert . (dot) to match all, same as (?s) ">reflex::convert_flag::dotall</a></code> </td><td>convert <code>.</code> (dot) to match all (match newline), same as <code>(?s)</code> </td></tr>
<tr>
<td><code><a class="el" href="namespacereflex_1_1convert__flag.html#a3c4525aa454d95740eac25f4bd0934c7" title="regex with multiline anchors ^ and $, same as (?m) ">reflex::convert_flag::multiline</a></code> </td><td>adds/asserts if <code>(?m)</code> is supported for multiline anchors <code>^</code> and <code>$</code> </td></tr>
</table>
<p>The following example enables Unicode matching by converting the regex pattern with the <code><a class="el" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39" title="convert . (dot), \s, \w, \l, \u, \S, \W, \L, \U to Unicode ">reflex::convert_flag::unicode</a></code> flag:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span> <span class="comment">// reflex::Matcher, reflex::Input, reflex::Pattern</span></div><div class="line"></div><div class="line"><span class="comment">// use a Matcher to check if sentence is in Greek:</span></div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> pattern(<a class="code" href="classreflex_1_1_matcher.html#a8862686f6d5680d4d0e5af146c4aed69">reflex::Matcher::convert</a>(<span class="stringliteral">&quot;[\\p{Greek}\\p{Zs}\\pP]+&quot;</span>, <a class="code" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39">reflex::convert_flag::unicode</a>));</div><div class="line"><span class="keywordflow">if</span> (<a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a>(pattern, sentence).matches())</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;This is Greek&quot;</span> &lt;&lt; std::endl;</div></div><!-- fragment --><p>The following example enables dotall mode to count the number of characters (including newlines) in the given <code>example</code> input:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span> <span class="comment">// reflex::BoostMatcher, reflex::Input, boost::regex</span></div><div class="line"></div><div class="line"><span class="comment">// construct a Boost.Regex matcher to count wide characters:</span></div><div class="line">std::string regex = <a class="code" href="classreflex_1_1_boost_matcher.html#ad1827615da2d51f4896e9c802b6463fc">reflex::BoostMatcher::convert</a>(<span class="stringliteral">&quot;(?su).&quot;</span>);</div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> boostmatcher(regex, example);</div><div class="line"><span class="keywordtype">size_t</span> n = std::distance(boostmatcher.scan.begin(), boostmatcher.scan.end());</div></div><!-- fragment --><p>Note that we could have used <code>"\\X"</code> instead to match any wide character without using the <code>(?su)</code> modifiers.</p>
<p>A converter throws a <code><a class="el" href="classreflex_1_1regex__error.html" title="Regex syntax error exceptions. ">reflex::regex_error</a></code> exception if conversion fails, for example when the regex syntax is invalid:</p>
<div class="fragment"><div class="line">std::string regex;</div><div class="line"><span class="keywordflow">try</span></div><div class="line">{</div><div class="line">  regex = <a class="code" href="classreflex_1_1_boost_matcher.html#ad1827615da2d51f4896e9c802b6463fc">reflex::BoostMatcher::convert</a>(argv[1], <a class="code" href="namespacereflex_1_1convert__flag.html#ab1c057a380f8673a13064c755652dc39">reflex::convert_flag::unicode</a>));</div><div class="line">}</div><div class="line"><span class="keywordflow">catch</span> (<a class="code" href="classreflex_1_1regex__error.html">reflex::regex_error</a>&amp; e)</div><div class="line">{</div><div class="line">  std::cerr &lt;&lt; e.what();</div><div class="line">  <span class="keywordflow">switch</span> (e.<a class="code" href="classreflex_1_1regex__error.html#a9db1cdde85bfe6f8f7807357addca122">code</a>())</div><div class="line">  {</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a293014262113a8affe812cdc1a155199">reflex::regex_error::mismatched_parens</a>:     std::cerr &lt;&lt; <span class="stringliteral">&quot;mismatched ( )&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#aa103b656bc8c84276caa3a3e479d5aed">reflex::regex_error::mismatched_braces</a>:     std::cerr &lt;&lt; <span class="stringliteral">&quot;mismatched { }&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a388b62b0f150f2c4e673a7326df98bf0">reflex::regex_error::mismatched_brackets</a>:   std::cerr &lt;&lt; <span class="stringliteral">&quot;mismatched [ ]&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a4da14a39e65667e922d4ac9251d3acf2">reflex::regex_error::mismatched_quotation</a>:  std::cerr &lt;&lt; <span class="stringliteral">&quot;mismatched \\Q...\\E quotation&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#aba10b497ace31b4e829961820f59ac1f">reflex::regex_error::empty_expression</a>:      std::cerr &lt;&lt; <span class="stringliteral">&quot;regex (sub)expression should not be empty&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#ab8d5ad921586a5821a5a6ea57f3ef7f3">reflex::regex_error::empty_class</a>:           std::cerr &lt;&lt; <span class="stringliteral">&quot;character class [...] is empty, e.g. [a&amp;&amp;[b]]&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#ac7c47c5a86f7f50f4128e4b31f3b8cd8">reflex::regex_error::invalid_class</a>:         std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid character class name&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a0a3dbc259753fe9c56bcb86ab22933f6">reflex::regex_error::invalid_class_range</a>:   std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid character class range, e.g. [Z-A]&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a3f67bcffa141ac3fd804dfe28a009165">reflex::regex_error::invalid_escape</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid escape character&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#abe00de8d9374f3b3528c181c3bbcee7c">reflex::regex_error::invalid_anchor</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid anchor or boundary&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a72584e74ccf493595d88f0ccc4b61f55">reflex::regex_error::invalid_repeat</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid repeat, e.g. {10,1}&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a1c01a511c833d13708a5607c7a6d810a">reflex::regex_error::invalid_quantifier</a>:    std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid lazy or possessive quantifier&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#af9454b097595a8c39fe0118cdbae60d0">reflex::regex_error::invalid_modifier</a>:      std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid (?ismux:) modifier&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a216ae66d5ecfe124a192a1e12fed6a00">reflex::regex_error::invalid_collating</a>:     std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid collating element&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#a90a346e348a466432d45f9251e3df4e5">reflex::regex_error::invalid_backreference</a>: std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid backreference&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="classreflex_1_1regex__error.html#ac3e3283b3859fae05b21655d8ee096a6">reflex::regex_error::invalid_syntax</a>:        std::cerr &lt;&lt; <span class="stringliteral">&quot;invalid regex syntax&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --><p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="regex-methods"></a>
Methods and iterators                                          </h2>
<p>The RE/flex abstract matcher, that every other RE/flex matcher inherits, provides four operations for matching with an instance of a regex engine:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>matches()</code> </td><td>returns nonzero if the input from begin to end matches </td></tr>
<tr>
<td><code>find()</code> </td><td>search input and return nonzero if a match was found </td></tr>
<tr>
<td><code>scan()</code> </td><td>scan input and return nonzero if input at current position matches </td></tr>
<tr>
<td><code>split()</code> </td><td>return nonzero for a split of the input at the next match </td></tr>
</table>
<p>These methods return a nonzero *"accept"* value for a match, meaning the <code>size_t accept()</code> value that corresponds to a group capture (or one if no groups are used). The methods are repeatable, where the last three return additional matches.</p>
<p>The <code>find</code>, <code>scan</code>, and <code>split</code> methods are also implemented as input iterators that apply filtering tokenization, and splitting:</p>
<table class="doxtable">
<tr>
<th>Iterator range </th><th>Acts as a </th><th>Iterates over  </th></tr>
<tr>
<td><code>find.begin()</code>...<code>find.end()</code> </td><td>filter </td><td>all matches </td></tr>
<tr>
<td><code>scan.begin()</code>...<code>scan.end()</code> </td><td>tokenizer </td><td>continuous matches </td></tr>
<tr>
<td><code>split.begin()</code>...<code>split.end()</code> </td><td>splitter </td><td>text between matches </td></tr>
</table>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-methods-matches"></a>
matches</h3>
<p>The <code>matches()</code> method returns a nonzero "accept" value (the <code>size_t accept()</code> group capture index value or the value 1 if no groups are used) if the given input from begin to the end matches the specified pattern.</p>
<p>For example, to match a UUID string with PCRE2:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line">std::string uuid = <span class="stringliteral">&quot;123e4567-e89b-12d3-a456-426655440000&quot;</span>;</div><div class="line"></div><div class="line"><span class="keywordflow">if</span> (<a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a>(<span class="stringliteral">&quot;[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}&quot;</span>, uuid).matches())</div><div class="line">  std::cout &lt;&lt; uuid &lt;&lt; <span class="stringliteral">&quot; is a string in UUID format&quot;</span> &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">123e4567-e89b-12d3-a456-426655440000 is a string in UUID format
</pre><p>The <code>matches()</code> method returns the group capture index that can be used as a selector. For example:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *food = <span class="stringliteral">&quot;hotdog&quot;</span>;</div><div class="line"></div><div class="line"><span class="keywordflow">switch</span> (<a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a>(<span class="stringliteral">&quot;(.*cat.*)|(.*dog.*)&quot;</span>, food).matches())</div><div class="line">{</div><div class="line">  <span class="keywordflow">case</span> 0: std::cout &lt;&lt; food &lt;&lt; <span class="stringliteral">&quot; has not cat or dog&quot;</span> &lt;&lt; std::endl;</div><div class="line">          <span class="keywordflow">break</span>;</div><div class="line">  <span class="keywordflow">case</span> 1: std::cout &lt;&lt; food &lt;&lt; <span class="stringliteral">&quot; has a cat&quot;</span> &lt;&lt; std::endl;</div><div class="line">          <span class="keywordflow">break</span>;</div><div class="line">  <span class="keywordflow">case</span> 2: std::cout &lt;&lt; food &lt;&lt; <span class="stringliteral">&quot; has a dog&quot;</span> &lt;&lt; std::endl;</div><div class="line">          <span class="keywordflow">break</span>;</div><div class="line">}</div></div><!-- fragment --><p>See also <a class="el" href="index.html#regex-methods-props">Properties of a match</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-methods-find"></a>
find</h3>
<p>The <code>find()</code> method and <code>find.begin()</code>...<code>find.end()</code> iterator range are used to search for a match in the given input.</p>
<p>The <code>find()</code> method returns a nonzero "accept" value (the <code>size_t accept()</code> group capture index value or the value 1 if no groups are used) for a match and zero otherwise.</p>
<p>For example, to find all words in a string with PCRE2:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; matcher.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">How
now
brown
cow
</pre><p>The iterator range <code>find.begin()</code>...<code>find.end()</code> serves as an input filter.</p>
<p>For example, in C++11 we can use a range-based loop to loop over matches using the <code>find</code> iterator:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="keywordflow">for</span> (<span class="keyword">auto</span>&amp; match : <a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a>(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>).find)</div><div class="line">  std::cout &lt;&lt; match.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>Iterators can be used with STL algorithms and other iterator functions. For example to count words in a string:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line">std::cout &lt;&lt; std::distance(matcher.find.begin(), matcher.find.end()) &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">4
</pre><p>The <code>find()</code> method returns the group capture index that can be used as a selector.</p>
<p>See also <a class="el" href="index.html#regex-methods-props">Properties of a match</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-methods-scan"></a>
scan</h3>
<p>The <code>scan()</code> method and <code>scan.begin()</code>...<code>scan.end()</code> iterator range are similar to <code>find</code> but generate continuous matches in the given input.</p>
<p>The <code>scan()</code> method returns a nonzero "accept" value (the <code>size_t accept()</code> group capture index value or the value 1 if no groups are used) for a match and zero otherwise.</p>
<p>For example, to scan for words, spacing, and punctuation in a sentence with PCRE2:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *tokens[4] = { <span class="stringliteral">&quot;n/a&quot;</span>, <span class="stringliteral">&quot;word&quot;</span>, <span class="stringliteral">&quot;space&quot;</span>, <span class="stringliteral">&quot;other&quot;</span> };</div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;(\\w+)|(\\s+)|(.)&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordtype">size_t</span> accept;</div><div class="line"><span class="keywordflow">while</span> ((accept = matcher.scan()) != 0)</div><div class="line">  std::cout &lt;&lt; tokens[accept] &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">word
space
word
space
word
space
word
other
</pre><p>The iterator range <code>scan.begin()</code>...<code>scan.end()</code> serves as an input tokenizer and produces continuous matches.</p>
<p>For example, tokenizing a string into a vector of numeric tokens:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;(\\w+)|(\\s+)|(.)&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line">std::vector&lt;size_t&gt; tokens(matcher.scan.begin(), matcher.scan.end());</div><div class="line">std::copy(tokens.begin(), tokens.end(), std::ostream_iterator&lt;size_t&gt;(std::cout, <span class="stringliteral">&quot; &quot;</span>));</div></div><!-- fragment --><p>When executed the code prints: </p><pre class="fragment">1 2 1 2 1 2 1 3
</pre><p>If the pattern does not match the input immediately succeeding a previous match, then the <code>scan()</code> method returns false and the iterator returns <code>scan.end()</code>. To determine if all input was scanned and end of input was reached, use the <code>at_end()</code> method, see <a class="el" href="index.html#regex-methods-props">Properties of a match</a>.</p>
<p>See also <a class="el" href="index.html#regex-methods-props">Properties of a match</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-methods-split"></a>
split</h3>
<p>The <code>split()</code> method and <code>split.begin()</code>...<code>split.end()</code> iterator range return text between matches in the given input.</p>
<p>The <code>split()</code> method returns a nonzero "accept" value (the <code>size_t accept()</code> group capture index value or the value 1 if no groups are used) of the matching text (that is, the text that comes after the split part) and zero otherwise.</p>
<p>When matches are adjacent then empty text is returned. Also the start of input and end of input return text that may be empty.</p>
<p>For example, to split text into words by matching non-words with PCRE2:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\W+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordflow">while</span> (matcher.split() != 0)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;&#39;&quot;</span> &lt;&lt; matcher.text() &lt;&lt; <span class="stringliteral">&quot;&#39;&quot;</span> &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">'How'
'now'
'brown'
'cow'
''
</pre><p>This produces five text splits where the last text is empty because the period at the end of the sentence matches the pattern and empty input remains.</p>
<p>The iterator range <code>split.begin()</code>...<code>split.end()</code> serves as an input splitter.</p>
<p>For example, to display the contents of a text file while normalizing spacing:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span> <span class="comment">// reflex::PCRE2Matcher, reflex::Input</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;\\s+&quot;</span>, fopen(<span class="stringliteral">&quot;abstract.txt&quot;</span>, <span class="stringliteral">&quot;r&quot;</span>));</div><div class="line"><span class="keywordflow">if</span> (matcher.in.file() != NULL)</div><div class="line">{</div><div class="line">  std::copy(matcher.split.begin(), matcher.split.end(), std::ostream_iterator&lt;std::string&gt;(std::cout, <span class="stringliteral">&quot; &quot;</span>));</div><div class="line">  fclose(matcher.in.file());</div><div class="line">}</div></div><!-- fragment --><dl class="section note"><dt>Note</dt><dd>The "accept" value of the last split is <code><a class="el" href="structreflex_1_1_abstract_matcher_1_1_const.html#a7c42ab60928bcc59f420876e741aae0b" title="accept() returns &quot;empty&quot; last split at end of input ">reflex::PCRE2Matcher::Const::EMPTY</a></code> to indicate that empty text after the split was found and matched. This special value is also returned by <code>accept()</code> and is also used with any other RE/flex matcher's <code>split</code> method.</dd></dl>
<p>See also <a class="el" href="index.html#regex-methods-props">Properties of a match</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-methods-props"></a>
Properties of a match</h3>
<p>To obtain properties of a match, use the following methods:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>accept()</code> </td><td>returns group capture index (or zero if not captured/matched) </td></tr>
<tr>
<td><code>text()</code> </td><td>returns <code>const char*</code> to 0-terminated text match (ends in <code>\0</code>) </td></tr>
<tr>
<td><code>str()</code> </td><td>returns <code>std::string</code> text match (preserves <code>\0</code>s) </td></tr>
<tr>
<td><code>wstr()</code> </td><td>returns <code>std::wstring</code> wide text match (converted from UTF-8) </td></tr>
<tr>
<td><code>chr()</code> </td><td>returns first 8-bit character of the text match (<code>str()[0]</code>) </td></tr>
<tr>
<td><code>wchr()</code> </td><td>returns first wide character of the text match (<code>wstr()[0]</code>) </td></tr>
<tr>
<td><code>pair()</code> </td><td>returns <code>std::pair&lt;size_t,std::string&gt;(accept(),str())</code> </td></tr>
<tr>
<td><code>wpair()</code> </td><td>returns <code>std::pair&lt;size_t,std::wstring&gt;(accept(),wstr())</code> </td></tr>
<tr>
<td><code>size()</code> </td><td>returns the length of the text match in bytes </td></tr>
<tr>
<td><code>wsize()</code> </td><td>returns the length of the match in number of wide characters </td></tr>
<tr>
<td><code>lines()</code> </td><td>returns the number of lines in the text match (&gt;=1) </td></tr>
<tr>
<td><code>columns()</code> </td><td>returns the number of columns of the text match (&gt;=0) </td></tr>
<tr>
<td><code>begin()</code> </td><td>returns <code>const char*</code> to non-0-terminated text match begin </td></tr>
<tr>
<td><code>end()</code> </td><td>returns <code>const char*</code> to non-0-terminated text match end </td></tr>
<tr>
<td><code>rest()</code> </td><td>returns <code>const char*</code> to 0-terminated rest of input </td></tr>
<tr>
<td><code>span()</code> </td><td>returns <code>const char*</code> to 0-terminated match enlarged to span the line </td></tr>
<tr>
<td><code>line()</code> </td><td>returns <code>std::string</code> line with the matched text as a substring </td></tr>
<tr>
<td><code>wline()</code> </td><td>returns <code>std::wstring</code> line with the matched text as a substring </td></tr>
<tr>
<td><code>more()</code> </td><td>tells the matcher to append the next match (when using <code>scan()</code>) </td></tr>
<tr>
<td><code>less(n)</code> </td><td>cuts <code>text()</code> to <code>n</code> bytes and repositions the matcher </td></tr>
<tr>
<td><code>lineno()</code> </td><td>returns line number of the match, starting at line 1 </td></tr>
<tr>
<td><code>columno()</code> </td><td>returns column number of the match, starting at 0 </td></tr>
<tr>
<td><code>lineno_end()</code> </td><td>returns ending line number of the match, starting at line 1 </td></tr>
<tr>
<td><code>columno_end()</code> </td><td>returns ending column number of the match, starting at 0 </td></tr>
<tr>
<td><code>border()</code> </td><td>returns byte offset from the start of the line of the match </td></tr>
<tr>
<td><code>first()</code> </td><td>returns input position of the first character of the match </td></tr>
<tr>
<td><code>last()</code> </td><td>returns input position + 1 of the last character of the match </td></tr>
<tr>
<td><code>at_bol()</code> </td><td>true if matcher reached the begin of a new line </td></tr>
<tr>
<td><code>at_bob()</code> </td><td>true if matcher is at the begin of input and no input consumed </td></tr>
<tr>
<td><code>at_end()</code> </td><td>true if matcher is at the end of input </td></tr>
<tr>
<td><code>[0]</code> </td><td>operator returns <code>std::pair&lt;const char*,size_t&gt;(begin(),size())</code> </td></tr>
<tr>
<td><code>[n]</code> </td><td>operator returns n'th capture <code>std::pair&lt;const char*,size_t&gt;</code> </td></tr>
</table>
<p>The <code>accept()</code> method returns nonzero for a succesful match, returning the group capture index. The RE/flex matcher engine <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> only recognizes group captures at the top level of the regex (i.e. among the top-level alternations), because it uses an efficient FSM for matching.</p>
<p>The <code>text()</code>, <code>str()</code>, and <code>wstr()</code> methods return the matched text. To get the first character of a match, use <code>chr()</code> or <code>wchr()</code>. The <code>chr()</code> and <code>wchr()</code> methods are much more efficient than <code>str()[0]</code> (or <code>text()[0]</code>) and <code>wstr()[0]</code>, respectively. Normally, a match cannot be empty unless option <code>"N"</code> is specified to explicitly initialize a matcher, see <a class="el" href="index.html#regex-boost">Boost matcher classes </a>, <a class="el" href="index.html#regex-pcre2">PCRE2 matcher classes </a>, and <a class="el" href="index.html#regex-matcher">The reflex::Matcher class </a>.</p>
<p>The <code>begin()</code>, <code>operator[0]</code>, and <code>operator[n]</code> return non-0-terminated strings. You must use <code>end()</code> with <code>begin()</code> to determine the span of the match. Basically, <code>text()</code> is the 0-terminated version of the string spanned by <code>begin()</code> to <code>end()</code>, where <code>end()</code> points the next character after the match, which means that <code>end()</code> = <code>begin()</code> + <code>size()</code>. Use the size of the capture returned by <code>operator[n]</code> to determine the end of the captured match.</p>
<p>The <code>lineno()</code> method returns the line number of the match, starting at line 1. The ending line number is <code>lineno_end()</code>, which is identical to the value of <code>lineno()</code> + <code>lines()</code> - 1.</p>
<p>The <code>columno()</code> method returns the column offset of a match from the start of the line, beginning at column 0. This method takes tab spacing and wide characters into account. The inclusive ending column number is given by <code>columno_end()</code>, which is equal or larger than <code>columno()</code> if the match does not span multiple lines. Otherwise, if the match spans multiple lines, <code>columno_end()</code> is the ending column of the match on the last matching line.</p>
<p>The starting byte offset of the match on a line is <code>border()</code> and the ending byte offset of the match is <code>border() + size() - 1</code>.</p>
<p>The <code>lines()</code> and <code>columns()</code> methods return the number of lines and columns matched, where <code>columns()</code> takes tab spacing and wide characters into account. If the match spans multiple lines, <code>columns()</code> counts columns over all lines, without counting the newline characters.</p>
<dl class="section note"><dt>Note</dt><dd>A wide character is counted as one, thus <code>columno()</code>, <code>columno_end()</code>, and <code>columns()</code> do not take the character width of full-width and combining Unicode characters into account. It is recommended to use the <code>wcwidth</code> function or <a href="https://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c">wcwidth.c</a> to determine Unicode character widths.</dd></dl>
<p>The <code>rest()</code> method returns the rest of the input character sequence as a 0-terminated <code>char*</code> string. This method buffers all remaining input to return the string.</p>
<p>The <code>span()</code> method enlarges the text matched to span the entire line and returns the matching line as a 0-terminated <code>char*</code> string without the <code>\n</code>.</p>
<p>The <code>line()</code> and <code>wline()</code> methods return the entire line as a (wide) string with the matched text as a substring. These methods can be used to obtain the context of a match.</p>
<dl class="section warning"><dt>Warning</dt><dd>The methods <code>span()</code>, <code>line()</code>, and <code>wline()</code> invalidate the previous <code>text()</code>, <code>begin()</code>, and <code>end()</code> string pointers. Call these methods again to retrieve the updated pointer or call <code>str()</code> or <code>wstr()</code> to obtain a string copy of the match: <div class="fragment"><div class="line"><span class="comment">// INCORRECT, because t is invalid after line():</span></div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *t = text();</div><div class="line">std::string s = line();</div><div class="line">std::cout &lt;&lt; t &lt;&lt; <span class="stringliteral">&quot; in &quot;</span> &lt;&lt; s &lt;&lt; std::endl;</div><div class="line"><span class="comment">// OK with line():</span></div><div class="line">std::string s = line();</div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *t = text();</div><div class="line">std::cout &lt;&lt; t &lt;&lt; <span class="stringliteral">&quot; in &quot;</span> &lt;&lt; s &lt;&lt; std::endl;</div><div class="line"><span class="comment">// OK with span():</span></div><div class="line">std::string t = str();</div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *s = span();</div><div class="line">std::cout &lt;&lt; t &lt;&lt; <span class="stringliteral">&quot; in &quot;</span> &lt;&lt; s &lt;&lt; std::endl;</div></div><!-- fragment --> The start of a line is truncated when the line is too long. The length of the line's contents before the pattern match on the line is restricted to 8KB, which is the size specified by <code><a class="el" href="structreflex_1_1_abstract_matcher_1_1_const.html#a6f7de58d1118b6f74f117a6b259359be" title="buffer growth factor, buffer is initially 2*BLOCK size, at least 256 ">reflex::AbstractMatcher::Const::BLOCK</a></code>. When this length is exceeded, the line's length before the match is truncated to 8KB. This ensures that pattern matching binary files or files with very long lines cannot cause memory allocation exceptions.</dd></dl>
<p>The <code>matcher().more()</code> method is used to create longer matches by stringing together consecutive matches in the input after scanning the input with the <code>scan()</code> method. When this method is invoked, the next match with <code>scan()</code> has its matched text prepended to it. The <code>matcher().more()</code> operation is often used in lexers and was introduced in Lex.</p>
<p>The <code>less(n)</code> method reduces the size of the matched text to <code>n</code> bytes. This method has no effect if <code>n</code> is larger than <code>size()</code>. The value of <code>n</code> should not be <code>0</code>. The <code>less(n)</code> operation is often used in lexers and was introduced in Lex.</p>
<p>The <code>first()</code> and <code>last()</code> methods return the position in the input stream of the match, counting in bytes from the start of the input at position 0. If the input stream is a wide character sequence, the UTF-8 positions are returned as a result of the internally-converted UTF-8 wide character input.</p>
<p>All methods take constant time to execute except for <code>str()</code>, <code>wstr()</code>, <code>pair()</code>, <code>wpair()</code>, <code>wsize()</code>, <code>lines()</code>, <code>columns()</code>, and <code>columno()</code> that require an extra pass over the matched text.</p>
<p>In addition, the following type casts of matcher objects and iterators may be used for convenience:</p>
<ul>
<li>Casting to <code>size_t</code> gives the matcher's <code>accept()</code> index.</li>
<li>Casting to <code>std::string</code> is the same as invoking <code>str()</code></li>
<li>Casting to <code>std::wstring</code> is the same as invoking <code>wstr()</code>.</li>
<li>Casting to <code>std::pair&lt;size_t,std::string&gt;</code> is the same as <code>pair()</code>.</li>
<li>Casting to <code>std::pair&lt;size_t,std::wstring&gt;</code> is the same as <code>wpair()</code>.</li>
</ul>
<p>The following example prints some of the properties of each match:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span> <span class="comment">// reflex::BoostMatcher, reflex::Input, boost::regex</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt;</div><div class="line">    <span class="stringliteral">&quot;accept: &quot;</span> &lt;&lt; matcher.accept() &lt;&lt;</div><div class="line">    <span class="stringliteral">&quot;text:   &quot;</span> &lt;&lt; matcher.text() &lt;&lt;</div><div class="line">    <span class="stringliteral">&quot;size:   &quot;</span> &lt;&lt; matcher.size() &lt;&lt;</div><div class="line">    <span class="stringliteral">&quot;line:   &quot;</span> &lt;&lt; matcher.lineno() &lt;&lt;</div><div class="line">    <span class="stringliteral">&quot;column: &quot;</span> &lt;&lt; matcher.columno() &lt;&lt;</div><div class="line">    <span class="stringliteral">&quot;first:  &quot;</span> &lt;&lt; matcher.first() &lt;&lt;</div><div class="line">    <span class="stringliteral">&quot;last:   &quot;</span> &lt;&lt; matcher.last() &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">accept: 1 text: How size: 3 line: 1 column: 0 first: 0 last: 3
accept: 1 text: now size: 3 line: 1 column: 4 first: 4 last: 7
accept: 1 text: brown size: 5 line: 1 column: 8 first: 8 last: 13
accept: 1 text: cow size: 3 line: 1 column: 14 first: 14 last: 17
</pre><p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-members"></a>
Public data members</h3>
<p>Four public data members of a matcher object are accesible:</p>
<table class="doxtable">
<tr>
<th>Variable </th><th>Usage  </th></tr>
<tr>
<td><code>in</code> </td><td>the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object used by the matcher </td></tr>
<tr>
<td><code>find</code> </td><td>the <code><a class="el" href="classreflex_1_1_abstract_matcher_1_1_operation.html" title="AbstractMatcher::Operation functor to match input to a pattern, also provides a (const) AbstractMatch...">reflex::AbstractMatcher::Operation</a></code> functor for searching </td></tr>
<tr>
<td><code>scan</code> </td><td>the <code><a class="el" href="classreflex_1_1_abstract_matcher_1_1_operation.html" title="AbstractMatcher::Operation functor to match input to a pattern, also provides a (const) AbstractMatch...">reflex::AbstractMatcher::Operation</a></code> functor for scanning </td></tr>
<tr>
<td><code>split</code> </td><td>the <code><a class="el" href="classreflex_1_1_abstract_matcher_1_1_operation.html" title="AbstractMatcher::Operation functor to match input to a pattern, also provides a (const) AbstractMatch...">reflex::AbstractMatcher::Operation</a></code> functor for splitting </td></tr>
</table>
<p>Normally only the <code>in</code> variable should be used which holds the current input object of the matcher. See <a class="el" href="index.html#regex-input">The Input class </a> for details.</p>
<p>The functors provide <code>begin()</code> and <code>end()</code> methods that return iterators and hold the necessary state information for the iterators. A functor invocation essentially invokes the corresponding method listed in <a class="el" href="index.html#regex-methods">Methods and iterators </a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-methods-pattern"></a>
Pattern methods</h3>
<p>To change a matcher's pattern or check if a pattern was assigned, you can use the following methods:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>pattern(p)</code> </td><td>set pattern to <code>p</code> (string regex or <code><a class="el" href="classreflex_1_1_pattern.html" title="Pattern class holds a regex pattern and its compiled FSM opcode table or code for the reflex::Matcher...">reflex::Pattern</a></code>) </td></tr>
<tr>
<td><code>has_pattern()</code> </td><td>true if the matcher has a pattern assigned to it </td></tr>
<tr>
<td><code>own_pattern()</code> </td><td>true if the matcher has a pattern to manage and delete </td></tr>
<tr>
<td><code>pattern()</code> </td><td>get the pattern object associated with the matcher </td></tr>
</table>
<p>The first method returns a reference to the matcher, so multiple method invocations may be chained together.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-methods-input"></a>
Input methods</h3>
<p>To assign a new input source to a matcher or set the input to buffered or interactive, you can use the following methods:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>input(i)</code> </td><td>set input to <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a> i</code> (string, stream, or <code>FILE*</code>) </td></tr>
<tr>
<td><code>buffer()</code> </td><td>buffer all input at once, returns true if successful </td></tr>
<tr>
<td><code>buffer(n)</code> </td><td>set the adaptive buffer size to <code>n</code> bytes to buffer input </td></tr>
<tr>
<td><code>interactive()</code> </td><td>sets buffer size to 1 for console-based (TTY) input </td></tr>
<tr>
<td><code>flush()</code> </td><td>flush the remaining input from the internal buffer </td></tr>
<tr>
<td><code>reset()</code> </td><td>resets the matcher, restarting it from the remaining input </td></tr>
<tr>
<td><code>reset(o)</code> </td><td>resets the matcher with new options string <code>o</code> ("A?N?T?") </td></tr>
</table>
<p>The first method returns a reference to the matcher, so multiple method invocations may be chained together.</p>
<p>The following methods may be used to read the input stream provided to a matcher directly, even when you use the matcher's search and match methods:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>input()</code> </td><td>returns next 8-bit char from the input, matcher then skips it </td></tr>
<tr>
<td><code>winput()</code> </td><td>returns next wide character from the input, matcher skips it </td></tr>
<tr>
<td><code>unput(c)</code> </td><td>put 8-bit char <code>c</code> back unto the stream, matcher then takes it </td></tr>
<tr>
<td><code>wunput(c)</code></td><td>put (wide) char <code>c</code> back unto the stream, matcher then takes it </td></tr>
<tr>
<td><code>peek()</code> </td><td>returns next 8-bit char from the input without consuming it </td></tr>
<tr>
<td><code>skip(c)</code> </td><td>skip input until character <code>c</code> (<code>char</code> or <code>wchar_t</code>) is consumed </td></tr>
<tr>
<td><code>skip(s)</code> </td><td>skip input until UTF-8 string <code>s</code> is consumed </td></tr>
<tr>
<td><code>rest()</code> </td><td>returns the remaining input as a non-NULL <code>char*</code> string </td></tr>
</table>
<p>The <code>input()</code>, <code>winput()</code>, and <code>peek()</code> methods return a non-negative character code and EOF (-1) when the end of input is reached.</p>
<p>A matcher reads from the specified input source using its virtual method <code>size_t get(char *s, size_t n)</code> that simply returns <code>in.get(s, n)</code>, that is, the result of the <code>reflex::Input::get(s, n)</code> method of the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object. The following protected methods may be overriden by a derived matcher class to customize reading:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>get(s, n)</code> </td><td>fill <code>s[0..n-1]</code> with next input, returns number of bytes read </td></tr>
<tr>
<td><code>wrap()</code> </td><td>returns false (may be overriden to wrap input after EOF) </td></tr>
</table>
<p>When a matcher reaches the end of input, it invokes the virtual method <code>wrap()</code> to check if more input is available. This method returns false by default, but this behavior may be changed by overriding <code>wrap()</code> to set a new input source and return <code>true</code>, for example:</p>
<div class="fragment"><div class="line"><span class="keyword">class </span>WrappedInputMatcher : <span class="keyword">public</span> <a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a> {</div><div class="line"> <span class="keyword">public</span>:</div><div class="line">  WrappedInputMatcher() : <a class="code" href="namespacereflex.html">reflex</a>::Matcher(), source_select(0)</div><div class="line">  { }</div><div class="line"> <span class="keyword">private</span>:</div><div class="line">  <span class="keyword">virtual</span> <span class="keywordtype">bool</span> <a class="code" href="classreflex_1_1_abstract_matcher.html#a046e0696b048d229ea2d77f5010c41e1">wrap</a>()</div><div class="line">  {</div><div class="line">    <span class="comment">// read a string, a file, and a string:</span></div><div class="line">    <span class="keywordflow">switch</span> (source_select++)</div><div class="line">    {</div><div class="line">      <span class="keywordflow">case</span> 0: in = <span class="stringliteral">&quot;Hello World!&quot;</span>;</div><div class="line">              <span class="keywordflow">return</span> <span class="keyword">true</span>;</div><div class="line">      <span class="keywordflow">case</span> 1: in = fopen(<span class="stringliteral">&quot;hello.txt&quot;</span>, <span class="stringliteral">&quot;r&quot;</span>);</div><div class="line">              <span class="keywordflow">return</span> in.file() != NULL;</div><div class="line">      <span class="keywordflow">case</span> 2: fclose(in.file());</div><div class="line">              in = <span class="stringliteral">&quot;Goodbye!&quot;</span>;</div><div class="line">              <span class="keywordflow">return</span> <span class="keyword">true</span>;</div><div class="line">    }</div><div class="line">    <span class="keywordflow">return</span> <span class="keyword">false</span>;</div><div class="line">  }</div><div class="line">  <span class="keywordtype">int</span> source_select;</div><div class="line">};</div></div><!-- fragment --><p>Note that the constructor in this example does not specify a pattern and input. To set a pattern for the matcher after its instantiation use the <code>pattern(p)</code> method. In this case the input does not need to be specified, which allows us to immediately force reading the sources of input that we assigned in our <code>wrap()</code> method.</p>
<p>For details of the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class, see <a class="el" href="index.html#regex-input">The Input class </a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="regex-input"></a>
The Input class                                                  </h2>
<p>A matcher may accept several types of input, but can only read from one input source at a time. Input to a matcher is represented by a single <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class instance that the matcher uses internally.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-input-assign"></a>
Assigning input</h3>
<p>An input object is constructed by specifying a string, a file, or a stream to read from. You can also reassign input to read from new input.</p>
<p>More specifically, you can pass a <code>std::string</code>, <code>char*</code>, <code>std::wstring</code>, <code>wchar_t*</code>, <code>FILE*</code>, or a <code>std::istream</code> to the constructor.</p>
<p>A <code>FILE*</code> file descriptor is a special case. The input object handles various file encodings. If a UTF Byte Order Mark (BOM) is detected then the UTF input will be normalized to UTF-8. When no UTF BOM is detected then the input is considered plain ASCII, binary, or UTF-8 and passed through unconverted. To override the file encoding when no UTF BOM was present, and normalize Latin-1, ISO-8859-1 through ISO-8859-15, CP 1250 through 1258, CP 437, CP 850, CP 858, KOI8, MACROMAN, EBCDIC, and other encodings to UTF-8, see <a class="el" href="index.html#regex-input-file">FILE encodings</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-input-strings"></a>
Input strings</h3>
<p>An input object constructed from an 8-bit string <code>char*</code> or <code>std::string</code> just passes the string to the matcher engine. The string should contain UTF-8 when Unicode patterns are used.</p>
<p>An input object constructed from a wide string <code>wchar_t*</code> or <code>std::wstring</code> translates the wide string to UTF-8 for matching, which effectively normalizes the input for matching with Unicode patterns. This conversion is illustrated below. The copyright symbol <code>©</code> with Unicode U+00A9 is matched against its UTF-8 sequence <code>C2 A9</code> of <code>©</code>:</p>
<div class="fragment"><div class="line"><span class="keywordflow">if</span> (<a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a>(<span class="stringliteral">&quot;©&quot;</span>, L<span class="stringliteral">&quot;©&quot;</span>).matches())</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;copyright symbol matches\n&quot;</span>;</div></div><!-- fragment --><p>To ensure that Unicode patterns in UTF-8 strings are grouped properly, use <a class="el" href="index.html#regex-convert">Regex converters </a>, for example as follows:</p>
<div class="fragment"><div class="line"><span class="keyword">static</span> <a class="code" href="classreflex_1_1_pattern.html">reflex::Pattern</a> CR(<a class="code" href="classreflex_1_1_matcher.html#a8862686f6d5680d4d0e5af146c4aed69">reflex::Matcher::convert</a>(<span class="stringliteral">&quot;(?u:\u{00A9})&quot;</span>));</div><div class="line"><span class="keywordflow">if</span> (<a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a>(CR, L<span class="stringliteral">&quot;©&quot;</span>).matches())</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;copyright symbol matches\n&quot;</span>;</div></div><!-- fragment --><p>Here we made the converted pattern static to avoid repeated conversion and construction overheads.</p>
<dl class="section note"><dt>Note</dt><dd>The <code>char*</code>, <code>wchar_t*</code>, and <code>std::wstring</code> strings cannot contain a <code>\0</code> (NUL) character and the first <code>\0</code> terminates matching. To match strings and binary input that contain <code>\0</code>, use <code>std::string</code> or <code>std::istringstream</code>.</dd></dl>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-input-streams"></a>
Input streams</h3>
<p>An input object constructed from a <code>std::istream</code> (or a derived class) just passes the input text to the matcher engine. The stream should contain ASCII and may contain UTF-8.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-input-file"></a>
FILE encodings</h3>
<p>File content specified with a <code>FILE*</code> file descriptor can be encoded in ASCII, binary, UTF-8/16/32, ISO-8859-1 through ISO-8859-15, CP 1250 through 1258, CP 437, CP 850, CP 858, or EBCDIC.</p>
<p>A <a href="www.unicode.org/faq/utf_bom.html">UTF Byte Order Mark (BOM)</a> is detected in the content of a file scanned by the matcher, which enables UTF-8 normalization of the input automatically.</p>
<p>Otherwise, if no file encoding is explicitly specified, the matcher expects raw UTF-8, ASCII, or plain binary by default. File formats can be decoded and translated to UTF-8 on the fly for matching by means of specifying encodings.</p>
<p>The current file encoding used by a matcher is obtained with the <code><a class="el" href="classreflex_1_1_input.html#a25682c7ebc7df115ade370d9d21ed888" title="Set encoding for FILE* input. ">reflex::Input::file_encoding()</a></code> method, which returns an <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html" title="Common file_encoding constants. ">reflex::Input::file_encoding</a></code> constant of type <code><a class="el" href="classreflex_1_1_input.html#a70e8550b0b1649b2cd49b1b872949cc8" title="Common file_encoding constants type. ">reflex::Input::file_encoding_type</a></code>:</p>
<table class="doxtable">
<tr>
<th>Constant </th><th>File encoding  </th></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#af213200d1cc11e672be420f1a4b61566" title="plain octets: 7-bit ASCII, 8-bit binary or UTF-8 without BOM detected ">reflex::Input::file_encoding::plain</a></code> </td><td>plain octets, ASCII/binary/UTF-8 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a20bd3b19b413e61afc3c665da1c64d0e" title="UTF-8 with BOM detected. ">reflex::Input::file_encoding::utf8</a></code> </td><td>UTF-8 (UTF BOM detected) </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a70bb4ad30b01c2b1b5061e105437d4de" title="UTF-16 big endian. ">reflex::Input::file_encoding::utf16be</a></code> </td><td>UTF-16 big endian (UTF BOM detected) </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#abe3c1db3aa8e7780eab85e201011f29e" title="UTF-16 little endian. ">reflex::Input::file_encoding::utf16le</a></code> </td><td>UTF-16 little endian (UTF BOM detected) </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a53577ddf6cf6f2d2f6bbf9630140273a" title="UTF-32 big endian. ">reflex::Input::file_encoding::utf32be</a></code> </td><td>UTF-32 big endian (UTF BOM detected) </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#af7ab91057731e548275a4eb63b6261ae" title="UTF-32 little endian. ">reflex::Input::file_encoding::utf32le</a></code> </td><td>UTF-32 little endian (UTF BOM detected) </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a70850a47577443173f3c31303612cdcf" title="ISO-8859-1, Latin-1. ">reflex::Input::file_encoding::latin</a></code> </td><td>ASCII with Latin-1, ISO-8859-1 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a533d3062b2eabd57d8d996f05a961c53" title="DOS CP 437. ">reflex::Input::file_encoding::cp437</a></code> </td><td>DOS CP 437 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a2c7cd01f85b15b2792a04021c415805d" title="DOS CP 850. ">reflex::Input::file_encoding::cp850</a></code> </td><td>DOS CP 850 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#aec2e4fc22c907af11338e829794c243f" title="DOS CP 858. ">reflex::Input::file_encoding::cp858</a></code> </td><td>DOS CP 858 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#ad80ba65b2eb9e237939b29a4758d8dda" title="EBCDIC. ">reflex::Input::file_encoding::ebcdic</a></code> </td><td>EBCDIC </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a1ecaa0c4f552a531c0eccfbe5ab006f1" title="Windows CP 1250. ">reflex::Input::file_encoding::cp1250</a></code> </td><td>Windows CP 1250 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#ac463724c06802db189b6aeb3b2ce0c04" title="Windows CP 1251. ">reflex::Input::file_encoding::cp1251</a></code> </td><td>Windows CP 1251 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a3ba20632cdb12411e69d29ac450037a3" title="Windows CP 1252. ">reflex::Input::file_encoding::cp1252</a></code> </td><td>Windows CP 1252 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#ac7ff7de99f3952909f66633f36c1fd25" title="Windows CP 1253. ">reflex::Input::file_encoding::cp1253</a></code> </td><td>Windows CP 1253 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#ac194ff983cbb3daafc103f52d220a912" title="Windows CP 1254. ">reflex::Input::file_encoding::cp1254</a></code> </td><td>Windows CP 1254 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#af559c1cfe8f5f60572c6ae4b715ec146" title="Windows CP 1255. ">reflex::Input::file_encoding::cp1255</a></code> </td><td>Windows CP 1255 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a8f7bd312945679f2d67dd0ea5eab44bf" title="Windows CP 1256. ">reflex::Input::file_encoding::cp1256</a></code> </td><td>Windows CP 1256 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#ab18e5f97380df73e18ff084987e41766" title="Windows CP 1257. ">reflex::Input::file_encoding::cp1257</a></code> </td><td>Windows CP 1257 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a3a978fa6caf666fd367adee792d2b799" title="Windows CP 1258. ">reflex::Input::file_encoding::cp1258</a></code> </td><td>Windows CP 1258 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a8a81b45fbf259ec1d13935375c35481d" title="ISO-8859-2, Latin-2. ">reflex::Input::file_encoding::iso8859_2</a></code> </td><td>ISO-8859-2 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a6ac76fc3cd8539231e452a80a02e631d" title="ISO-8859-3, Latin-3. ">reflex::Input::file_encoding::iso8859_3</a></code> </td><td>ISO-8859-3 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a27a8b9e0e2cc441036d4b4a926485e8a" title="ISO-8859-4, Latin-4. ">reflex::Input::file_encoding::iso8859_4</a></code> </td><td>ISO-8859-4 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a53ed11fde3562f4d5c53581bbe2d7608" title="ISO-8859-5, Cyrillic. ">reflex::Input::file_encoding::iso8859_5</a></code> </td><td>ISO-8859-5 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a0f5cba1a52a8e5d484e0613c730be931" title="ISO-8859-6, Arabic. ">reflex::Input::file_encoding::iso8859_6</a></code> </td><td>ISO-8859-6 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#aaf9516e1d489edd6b7aae831b0d27856" title="ISO-8859-7, Greek. ">reflex::Input::file_encoding::iso8859_7</a></code> </td><td>ISO-8859-7 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a5035cf2b775ab05446b61c84558697c5" title="ISO-8859-8, Hebrew. ">reflex::Input::file_encoding::iso8859_8</a></code> </td><td>ISO-8859-8 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a90d5c3ee945f132789a2805656547938" title="ISO-8859-9, Latin-5. ">reflex::Input::file_encoding::iso8859_9</a></code> </td><td>ISO-8859-9 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#ac8d51f0373dbf4e65c404e6e08cc4ce9" title="ISO-8859-10, Latin-6. ">reflex::Input::file_encoding::iso8859_10</a></code> </td><td>ISO-8859-10 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a885b87ce71146c49f8eceef4d939cef5" title="ISO-8859-11, Thai. ">reflex::Input::file_encoding::iso8859_11</a></code> </td><td>ISO-8859-11 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a56a15bc6091f84a47083719c1ac84f4b" title="ISO-8859-13, Latin-7. ">reflex::Input::file_encoding::iso8859_13</a></code> </td><td>ISO-8859-13 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a2633104718050d152b044160e79b1396" title="ISO-8859-14, Latin-8. ">reflex::Input::file_encoding::iso8859_14</a></code> </td><td>ISO-8859-14 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a446455dea74b6e8995036ff3cc2dab82" title="ISO-8859-15, Latin-9. ">reflex::Input::file_encoding::iso8859_15</a></code> </td><td>ISO-8859-15 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a96ecc25ba97a877c5d03ec3f32b44464" title="ISO-8859-16. ">reflex::Input::file_encoding::iso8859_16</a></code> </td><td>ISO-8859-16 </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a5e9417330585b795cb0814de2f751f6b" title="Macintosh Roman with CR to LF translation. ">reflex::Input::file_encoding::macroman</a></code> </td><td>Macintosh Roman + CR to LF translation </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a8f1795ff2b9505ff8bc114721f40b336" title="KOI8-R. ">reflex::Input::file_encoding::koi8_r</a></code> </td><td>KOI8-R </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#afe9401d703dd4e82d63e694118c2bf39" title="KOI8-U. ">reflex::Input::file_encoding::koi8_u</a></code> </td><td>KOI8-U </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#ab7453c75ea45732498a8cbd18e38aa7c" title="KOI8-RU. ">reflex::Input::file_encoding::koi8_ru</a></code> </td><td>KOI8-RU </td></tr>
<tr>
<td><code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a76935cbd4a24d36af7e3ebdd5772aeb9" title="custom code page ">reflex::Input::file_encoding::custom</a></code> </td><td>user-defined custom code page </td></tr>
</table>
<p>To set the file encoding when assigning a file to read with <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code>, use <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input(file, enc)</a></code> with one of the encoding constants shown in the table.</p>
<p>For example, use <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a70850a47577443173f3c31303612cdcf" title="ISO-8859-1, Latin-1. ">reflex::Input::file_encoding::latin</a></code> to override the encoding when the file contains ISO-8859-1. This way you can match its content using Unicode patterns (matcher engines internally normalizes ISO-8859-1 to UTF-8):</p>
<div class="fragment"><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> input(stdin, <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a70850a47577443173f3c31303612cdcf">reflex::Input::file_encoding::latin</a>);</div><div class="line"><a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a> matcher(pattern, input);</div></div><!-- fragment --><p>This sets the standard input encoding to ISO-8859-1, but only if no UTF BOM was detected on the standard input, because the UTF encoding of a <code>FILE*</code> that starts with a UTF BOM cannot be overruled.</p>
<p>To define a custom code page to translate files, define a code page table with 256 entries that maps each 8-bit input character to a 16-bit Unicode character (UCS-2). Then use <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a76935cbd4a24d36af7e3ebdd5772aeb9" title="custom code page ">reflex::Input::file_encoding::custom</a></code> with a pointer to your code page to construct an input object. For example:</p>
<div class="fragment"><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> <span class="keywordtype">unsigned</span> <span class="keywordtype">short</span> CP[256] = {</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">   32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,</div><div class="line">   48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,</div><div class="line">   64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,</div><div class="line">   80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95,</div><div class="line">   96, 97, 98, 99,100,101,102,103,104,105,106,107,108,109,110,111,</div><div class="line">  112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">   32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,</div><div class="line">};</div><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> input(stdin, <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a76935cbd4a24d36af7e3ebdd5772aeb9">reflex::Input::file_encoding::custom</a>, CP);</div><div class="line"><a class="code" href="classreflex_1_1_matcher.html">reflex::Matcher</a> matcher(pattern, input);</div></div><!-- fragment --><p>This example translates all control characters and characters above 127 to spaces before matching.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-input-properties"></a>
Input properties</h3>
<p>To obtain the properties of an input source use the following methods:</p>
<table class="doxtable">
<tr>
<th>Method </th><th>Result  </th></tr>
<tr>
<td><code>size()</code> </td><td>size in bytes of the remaining input, zero when EOF or unknown </td></tr>
<tr>
<td><code>good()</code> </td><td>input is available to read (no error and not EOF) </td></tr>
<tr>
<td><code>eof()</code> </td><td>end of input (but use only <code>at_end()</code> with matchers!) </td></tr>
<tr>
<td><code>cstring()</code> </td><td>the current <code>const char*</code> (of a <code>std::string</code>) or NULL </td></tr>
<tr>
<td><code>wstring()</code> </td><td>the current <code>const wchar_t*</code> (of a <code>std::wstring</code>) or NULL </td></tr>
<tr>
<td><code>file()</code> </td><td>the current <code>FILE*</code> file descriptor or NULL </td></tr>
<tr>
<td><code>istream()</code> </td><td>a <code>std::istream*</code> pointer to the current stream object or NULL </td></tr>
</table>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-input-streambuf"></a>
Input streambuf</h3>
<p>We can use a <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object as a <code>std::streambuf</code> and pass it to a <code>std::istream</code>. This is useful when a <code>std::istream</code> is required where a <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object cannot be directly used. The <code>std::istream</code> automatically normalizes the input to UTF-8 using the underlying <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object. For example:</p>
<div class="fragment"><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> input(...);            <span class="comment">// create an Input object for some given input</span></div><div class="line"><a class="code" href="classreflex_1_1_input_1_1streambuf.html">reflex::Input::streambuf</a> buf(input); <span class="comment">// create a streambuf</span></div><div class="line">std::istream is(&amp;buf);</div><div class="line"><span class="keywordflow">if</span> (is.good())</div><div class="line">{</div><div class="line">  <span class="comment">// read the stream</span></div><div class="line">}</div></div><!-- fragment --><p>The <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object may be created from strings, wide strings, streams, and <code>FILE*</code> values. These are readable as a <code>std::istream</code> via <code><a class="el" href="classreflex_1_1_input_1_1streambuf.html" title="Stream buffer for reflex::Input, derived from std::streambuf. ">reflex::Input::streambuf</a></code> that returns normalized UTF-8 characters. For <code>FILE*</code> values we can specify <a class="el" href="index.html#regex-input-file">FILE encodings</a> to normalize the encoded input to UTF-8.</p>
<p>Keep in mind that adding a <code>std::istream</code> with <code><a class="el" href="classreflex_1_1_input_1_1streambuf.html" title="Stream buffer for reflex::Input, derived from std::streambuf. ">reflex::Input::streambuf</a></code> layer on top of the efficient <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class will impact file reading performance, especially because <code><a class="el" href="classreflex_1_1_input_1_1streambuf.html" title="Stream buffer for reflex::Input, derived from std::streambuf. ">reflex::Input::streambuf</a></code> is unbuffered (despite its name). When performance is important, use the buffered version <code><a class="el" href="classreflex_1_1_buffered_input_1_1streambuf.html" title="Buffered stream buffer for reflex::Input, derived from std::streambuf. ">reflex::BufferedInput::streambuf</a></code>:</p>
<div class="fragment"><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> input(...);                    <span class="comment">// create an Input object for some given input</span></div><div class="line"><a class="code" href="classreflex_1_1_buffered_input_1_1streambuf.html">reflex::BufferedInput::streambuf</a> buf(input); <span class="comment">// create a buffered streambuf</span></div><div class="line">std::istream is(&amp;buf);</div><div class="line"><span class="keywordflow">if</span> (is.good())</div><div class="line">{</div><div class="line">  <span class="comment">// read the stream</span></div><div class="line">}</div></div><!-- fragment --><p>Because the buffered vesion reads ahead to fill its buffer, the buffered version may not be suitable for interactive input.</p>
<p>See also <a class="el" href="index.html#regex-input-dosstreambuf">DOS CRLF newlines</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3><a class="anchor" id="regex-input-dosstreambuf"></a>
DOS CRLF newlines</h3>
<p>DOS files and other DOS or Windows input sources typically end lines with CRLF byte pairs, see <a class="el" href="index.html#crlf">Handling DOS CRLF newlines </a>. To automatically replace CRLF by LF when reading files in "binary mode" you can use the <code><a class="el" href="classreflex_1_1_input_1_1dos__streambuf.html" title="Stream buffer for reflex::Input to read DOS files, replaces CRLF by LF, derived from std::streambuf...">reflex::Input::dos_streambuf</a></code> class to construct a <code>std::istream</code> object. This normalized stream can then be used as input to a RE/flex scanner or to a regex matcher:</p>
<div class="fragment"><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> input(...);                <span class="comment">// create an Input object for some given input</span></div><div class="line"><a class="code" href="classreflex_1_1_input_1_1dos__streambuf.html">reflex::Input::dos_streambuf</a> buf(input); <span class="comment">// create a dos_streambuf</span></div><div class="line">std::istream is(&amp;buf);</div><div class="line"><span class="keywordflow">if</span> (is.good())</div><div class="line">{</div><div class="line">  <span class="comment">// read the stream</span></div><div class="line">}</div></div><!-- fragment --><p>Once the stream object is created it can be used to create a new input object for a RE/flex scanner, for example:</p>
<div class="fragment"><div class="line"><span class="keywordflow">if</span> (is.good())</div><div class="line">{</div><div class="line">  Lexer lexer(is); <span class="comment">// create a lexer</span></div><div class="line">  lexer.lex();     <span class="comment">// scan the normalized input (DOS CRLF -&gt; LF)</span></div><div class="line">}</div></div><!-- fragment --><p>or for a regex matcher that uses Boost.Regex or PCRE2:</p>
<div class="fragment"><div class="line"><span class="keywordflow">if</span> (is.good())</div><div class="line">{</div><div class="line">  <a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, is); <span class="comment">// read normalized input (DOS CRLF -&gt; LF)</span></div><div class="line">  <span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">    std::cout &lt;&lt; <span class="stringliteral">&quot;Found &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div><div class="line">}</div></div><!-- fragment --><p>Note that when the input is a <code>FILE*</code>, CRLF pairs are replaced by LF and UTF-16/32 encodings are automatically normalized to UTF-8 (when a UTF BOM is present in the file or you can specify <a class="el" href="index.html#regex-input-file">FILE encodings</a>).</p>
<dl class="section warning"><dt>Warning</dt><dd>The <code><a class="el" href="classreflex_1_1_input.html#a4d1e21f3356426598bc6dc139812de31" title="Get the size of the input character sequence in number of ASCII/UTF-8 bytes (zero if size is not dete...">reflex::Input::size</a></code> method returns the number of bytes available that includes CRLF pairs. The actual number of bytes read may be smaller after replacing CRLF by LF.</dd></dl>
<p>When performance is important, use the buffered version <code><a class="el" href="classreflex_1_1_buffered_input_1_1dos__streambuf.html" title="Buffered stream buffer for reflex::Input to read DOS files, replaces CRLF by LF, derived from std::st...">reflex::BufferedInput::dos_streambuf</a></code>:</p>
<div class="fragment"><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> input(...);                        <span class="comment">// create an Input object for some given input</span></div><div class="line"><a class="code" href="classreflex_1_1_buffered_input_1_1dos__streambuf.html">reflex::BufferedInput::dos_streambuf</a> buf(input); <span class="comment">// create a buffered dos_streambuf</span></div><div class="line">std::istream is(&amp;buf);</div><div class="line"><span class="keywordflow">if</span> (is.good())</div><div class="line">{</div><div class="line">  <span class="comment">// read the stream</span></div><div class="line">}</div></div><!-- fragment --><p>Because the buffered vesion reads ahead to fill its buffer, the buffered version may not be suitable for interactive input:</p>
<p>See also <a class="el" href="index.html#regex-input-streambuf">Input streambuf</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="regex-examples"></a>
Examples                                                      </h2>
<p>This section includes several examples to demonstrate the concepts discussed.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 1</h3>
<p>This example illustrates the <code>find</code> and <code>split</code> methods and iterators with a RE/flex <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> and a <code><a class="el" href="classreflex_1_1_boost_matcher.html" title="Boost matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::BoostMatcher</a></code> using a C++11 range-based loop:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span></div><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacereflex.html">reflex</a>;</div><div class="line"></div><div class="line"><span class="comment">// some random text (with extra spacing)</span></div><div class="line">std::string example(<span class="stringliteral">&quot;Monty\n Python&#39;s    Flying  Circus&quot;</span>);</div><div class="line"></div><div class="line"><span class="comment">// construct a fixed pattern that is case insensitive</span></div><div class="line"><span class="keyword">static</span> <a class="code" href="classreflex_1_1_pattern.html">Pattern</a> pattern(<span class="stringliteral">&quot;monty|python&quot;</span>, <span class="stringliteral">&quot;i&quot;</span>);</div><div class="line"></div><div class="line"><span class="comment">// construct a matcher to search the example text</span></div><div class="line"><a class="code" href="classreflex_1_1_matcher.html">Matcher</a> matcher(pattern, example);</div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">  std::cout &lt;&lt; matcher.text()</div><div class="line">            &lt;&lt; <span class="stringliteral">&quot; at &quot;</span> &lt;&lt; matcher.lineno() &lt;&lt; <span class="stringliteral">&quot;,&quot;</span> &lt;&lt; matcher.columno()</div><div class="line">            &lt;&lt; <span class="stringliteral">&quot; spans &quot;</span> &lt;&lt; matcher.first() &lt;&lt; <span class="stringliteral">&quot;..&quot;</span> &lt;&lt; matcher.last()</div><div class="line">            &lt;&lt; std::endl;</div><div class="line"></div><div class="line"><span class="comment">// construct a Boost.Regex matcher to count the words (all non-spaces &quot;\S+&quot;)</span></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">BoostMatcher</a> boostmatcher(<span class="stringliteral">&quot;\\S+&quot;</span>, example);</div><div class="line">boostmatcher.buffer(); <span class="comment">// because Boost.Regex partial_match is broken!</span></div><div class="line">std::cout &lt;&lt; std::distance(boostmatcher.find.begin(), boostmatcher.find.end())</div><div class="line">          &lt;&lt; <span class="stringliteral">&quot; words&quot;</span></div><div class="line">          &lt;&lt; std::endl;</div><div class="line"></div><div class="line"><span class="comment">// reuse the BoostMatcher with a C++11 range-based loop to split the example text on space &quot;\s+&quot;:</span></div><div class="line"><span class="keywordflow">for</span> (<span class="keyword">auto</span>&amp; split : boostmatcher.pattern(<span class="stringliteral">&quot;\\s+&quot;</span>).input(example).split)</div><div class="line">  std::cout &lt;&lt; split.text() &lt;&lt; <span class="stringliteral">&quot; &quot;</span>;</div><div class="line">std::cout &lt;&lt; std::endl;</div><div class="line"></div><div class="line"><span class="comment">// reuse the BoostMatcher to split on whitespace using an iterator, sort into a set, and print</span></div><div class="line">boostmatcher.input(example).buffer();</div><div class="line">std::set&lt;std::string&gt; words(boostmatcher.split.begin(), boostmatcher.split.end());</div><div class="line">std::copy(words.begin(), words.end(), std::ostream_iterator&lt;std::string&gt;(std::cout, <span class="stringliteral">&quot; &quot;</span>));</div><div class="line">std::cout &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">Monty at 1,0 spans 0..5
Python at 2,1 spans 7..13
4 words
Monty Python's Flying Circus
Circus Flying Monty Python's
</pre><p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 2</h3>
<p>This example shows how a URL can be matched by using two patterns: one pattern to extract the host:port/path parts and another pattern to extract the query string key-value pairs in a loop. A PCRE2 matcher or a Boost.Regex matcher may be used, since both support group captures.</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span></div><div class="line"><span class="preprocessor">#include &lt;iostream&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacereflex.html">reflex</a>;</div><div class="line"></div><div class="line"><span class="keyword">const</span> <span class="keywordtype">char</span> *URL = <span class="stringliteral">&quot;...&quot;</span>;</div><div class="line"></div><div class="line"><span class="comment">// match URL host:port/path using group captures for these</span></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">BoostMatcher</a> re(<span class="stringliteral">&quot;https?://([^:/]*):?(\\d*)/?([^?#]*)&quot;</span>, URL);</div><div class="line"></div><div class="line"><span class="keywordflow">if</span> (re.scan())</div><div class="line">{</div><div class="line">  <span class="comment">// found a partial match at start, now check if we have a host</span></div><div class="line">  <span class="keywordflow">if</span> (re[1].first != NULL)</div><div class="line">  {</div><div class="line">    std::string host(re[1].first, re[1].second);</div><div class="line">    std::cout &lt;&lt; <span class="stringliteral">&quot;host: &quot;</span> &lt;&lt; host &lt;&lt; std::endl;</div><div class="line"></div><div class="line">    <span class="comment">// check of we have a port</span></div><div class="line">    <span class="keywordflow">if</span> (re[2].first != NULL &amp;&amp; re[2].second != 0)</div><div class="line">    {</div><div class="line">      std::string port(re[2].first, re[2].second);</div><div class="line">      std::cout &lt;&lt; <span class="stringliteral">&quot;port: &quot;</span> &lt;&lt; port &lt;&lt; std::endl;</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="comment">// check of we have a path</span></div><div class="line">    <span class="keywordflow">if</span> (re[3].first != NULL &amp;&amp; re[3].second != 0)</div><div class="line">    {</div><div class="line">      std::string path(re[3].first, re[3].second);</div><div class="line">      std::cout &lt;&lt; <span class="stringliteral">&quot;path: &quot;</span> &lt;&lt; path &lt;&lt; std::endl;</div><div class="line">    }</div><div class="line">  }</div><div class="line"></div><div class="line">  <span class="comment">// check if we have a query string</span></div><div class="line">  <span class="keywordflow">if</span> (re.input() == <span class="charliteral">&#39;?&#39;</span>)</div><div class="line">  {</div><div class="line">    <span class="comment">// now switch patterns to match the query string</span></div><div class="line">    re.pattern(<span class="stringliteral">&quot;([^=&amp;]*)=?([^&amp;]*)&amp;?&quot;</span>);</div><div class="line">    <span class="keywordflow">while</span> (re.scan())</div><div class="line">      std::cout &lt;&lt;</div><div class="line">        <span class="stringliteral">&quot;query key: &quot;</span> &lt;&lt; std::string(re[1].first, re[1].second) &lt;&lt;</div><div class="line">        <span class="stringliteral">&quot;, value: &quot;</span> &lt;&lt; std::string(re[2].first, re[2].second) &lt;&lt; std::endl;</div><div class="line">  }</div><div class="line">  <span class="keywordflow">else</span> <span class="keywordflow">if</span> (!re.at_end())</div><div class="line">  {</div><div class="line">    <span class="comment">// not a query string and not the end, we expect an # anchor</span></div><div class="line">    std::cout &lt;&lt; <span class="stringliteral">&quot;anchor: &quot;</span> &lt;&lt; re.rest() &lt;&lt; std::endl;</div><div class="line">  }</div><div class="line">}</div><div class="line"><span class="keywordflow">else</span></div><div class="line">{</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Error, not a http/s URL: &quot;</span> &lt;&lt; re.rest() &lt;&lt; std::endl;</div><div class="line">}</div></div><!-- fragment --><p>See also Example 8 below for a more powerful URL pattern matcher.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 3</h3>
<p>This example shows how input can be reassigned in each iteration of a loop that matches wide strings against a word pattern <code>\w+</code>:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacereflex.html">reflex</a>;</div><div class="line"></div><div class="line"><span class="comment">// four words</span></div><div class="line"><span class="keyword">const</span> <span class="keywordtype">wchar_t</span> *words[] = { L<span class="stringliteral">&quot;Monty&quot;</span>, L<span class="stringliteral">&quot;Python&#39;s&quot;</span>, L<span class="stringliteral">&quot;Flying&quot;</span>, L<span class="stringliteral">&quot;Circus&quot;</span> };</div><div class="line"></div><div class="line"><span class="comment">// construct a PCRE2 matcher for words, given empty input initially</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">PCRE2Matcher</a> wordmatcher(<span class="stringliteral">&quot;\\w+&quot;</span>, Input());</div><div class="line"></div><div class="line"><span class="comment">// check if each string in words[] is a word</span></div><div class="line"><span class="keywordflow">for</span> (<span class="keywordtype">int</span> i = 0; i &lt; 4; i++)</div><div class="line">  <span class="keywordflow">if</span> (wordmatcher.input(words[i]).matches())</div><div class="line">    std::cout &lt;&lt; wordmatcher.text() &lt;&lt; <span class="stringliteral">&quot;, &quot;</span>;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">Monty, Flying, Circus, 
</pre><p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 4</h3>
<p>This example counts the number of words, lines, and chars from the <code>std::cin</code> stream:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="stdmatcher_8h.html">reflex/stdmatcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacereflex.html">reflex</a>;</div><div class="line"></div><div class="line"><span class="comment">// construct a std::regex matcher like the wc command (a word is a series of nonspaces)</span></div><div class="line"><a class="code" href="classreflex_1_1_std_matcher.html">StdMatcher</a> word(<span class="stringliteral">&quot;\\S+&quot;</span>, std::cin);</div><div class="line"></div><div class="line"><span class="keywordtype">size_t</span> words = std::distance(word.find.begin(), word.find.end());</div><div class="line"><span class="keywordtype">size_t</span> lines = word.lineno() - 1;</div><div class="line"><span class="keywordtype">size_t</span> chars = word.last();</div><div class="line"></div><div class="line">std::cout &lt;&lt; lines &lt;&lt; <span class="stringliteral">&quot; &quot;</span> &lt;&lt; words &lt;&lt; <span class="stringliteral">&quot; &quot;</span> &lt;&lt; chars &lt;&lt; std::endl;</div></div><!-- fragment --><p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 5</h3>
<p>This example tokenizes a string by grouping the subpatterns in a regex and by using the group index of the capture obtained with <code>accept()</code> in a C++11 range-based loop:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacereflex.html">reflex</a>;</div><div class="line"></div><div class="line"><span class="keyword">static</span> <a class="code" href="classreflex_1_1_pattern.html">Pattern</a> pattern =</div><div class="line">  <span class="stringliteral">&quot;(\\w*cat\\w*)|&quot;</span> <span class="comment">// 1st group = token 1</span></div><div class="line">  <span class="stringliteral">&quot;(\\w*dog\\w*)|&quot;</span> <span class="comment">// 2nd group = token 2</span></div><div class="line">  <span class="stringliteral">&quot;(\\w+)|&quot;</span>        <span class="comment">// 3rd group = token 3</span></div><div class="line">  <span class="stringliteral">&quot;(.)&quot;</span>;           <span class="comment">// 4th group = token 4</span></div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_matcher.html">Matcher</a> tokenizer(pattern, <span class="stringliteral">&quot;cats love hotdogs!&quot;</span>);</div><div class="line"></div><div class="line"><span class="keywordflow">for</span> (<span class="keyword">auto</span>&amp; token : tokenizer.scan)</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Token = &quot;</span> &lt;&lt; token.accept()</div><div class="line">            &lt;&lt; <span class="stringliteral">&quot;: matched &#39;&quot;</span> &lt;&lt; token.text() &lt;&lt; <span class="stringliteral">&quot;&#39;&quot;</span></div><div class="line">            &lt;&lt; <span class="stringliteral">&quot; with &#39;&quot;</span> &lt;&lt; token.pattern()[token.accept()] &lt;&lt; <span class="stringliteral">&quot;&#39;\n&quot;</span>;</div><div class="line">assert(tokenizer.at_end());</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">Token = 1: matched 'cats' with '(\\w*cat\\w*)'
Token = 4: matched ' ' with '(.)'
Token = 3: matched 'love' with '(\\w+)'
Token = 4: matched ' ' with '(.)'
Token = 2: matched 'hotdogs' with '(\\w*dog\\w*)'
Token = 4: matched '!' with '(.)'
</pre><p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 6</h3>
<p>This example reads a file with embedded credit card numbers to extract. The numbers are sorted into five sets for each type of major credit card:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="matcher_8h.html">reflex/matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacereflex.html">reflex</a>;</div><div class="line"></div><div class="line"><span class="keyword">static</span> <a class="code" href="classreflex_1_1_pattern.html">Pattern</a> card_patterns =</div><div class="line">  <span class="stringliteral">&quot;(?# MasterCard)(5[1-5]\\d{14})|&quot;</span>                   <span class="comment">// 1st group = MC</span></div><div class="line">  <span class="stringliteral">&quot;(?# Visa)(4\\d{12}(?:\\d{3})?)|&quot;</span>                   <span class="comment">// 2nd group = VISA</span></div><div class="line">  <span class="stringliteral">&quot;(?# AMEX)(3[47]\\d{13})|&quot;</span>                          <span class="comment">// 3rd group = AMEX</span></div><div class="line">  <span class="stringliteral">&quot;(?# Discover)(6011\\d{14})|&quot;</span>                       <span class="comment">// 4th group = Discover</span></div><div class="line">  <span class="stringliteral">&quot;(?# Diners Club)((?:30[0-5]|36\\d|38\\d)\\d{11})&quot;</span>; <span class="comment">// 5th group = Diners</span></div><div class="line"></div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> <span class="keywordtype">char</span> *card_data =</div><div class="line">  <span class="stringliteral">&quot;mark 5212345678901234\n&quot;</span></div><div class="line">  <span class="stringliteral">&quot;vinny 4123456789012\n&quot;</span></div><div class="line">  <span class="stringliteral">&quot;victor 4123456789012345\n&quot;</span></div><div class="line">  <span class="stringliteral">&quot;amy 371234567890123\n&quot;</span></div><div class="line">  <span class="stringliteral">&quot;dirk 601112345678901234\n&quot;</span></div><div class="line">  <span class="stringliteral">&quot;doc 38812345678901 end\n&quot;</span>;</div><div class="line"></div><div class="line">std::set&lt;std::string&gt; cards[5];</div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_matcher.html">Matcher</a> matcher(card_patterns, card_data);</div><div class="line"><span class="keywordflow">for</span> (<a class="code" href="classreflex_1_1_abstract_matcher.html#a7237c1fb8ab2a2283117739e6516c31f">Matcher::iterator</a> match = matcher.find.begin(); match != matcher.find.end(); ++match)</div><div class="line">  cards[match.accept() - 1].insert(match.text());</div><div class="line"></div><div class="line"><span class="keywordflow">for</span> (<span class="keywordtype">int</span> i = 0; i &lt; 5; ++i)</div><div class="line">  <span class="keywordflow">for</span> (std::set&lt;std::string&gt;::const_iterator j = cards[i].begin(); j != cards[i].end(); ++j)</div><div class="line">    std::cout &lt;&lt; i &lt;&lt; <span class="stringliteral">&quot;: &quot;</span> &lt;&lt; *j &lt;&lt; std::endl;</div></div><!-- fragment --><p>When executed this code prints: </p><pre class="fragment">0: 5212345678901234
1: 4123456789012
1: 4123456789012345
2: 371234567890123
3: 601112345678901234
4: 38812345678901
</pre><p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 7</h3>
<p>The RE/flex matcher engine <code><a class="el" href="classreflex_1_1_matcher.html" title="RE/flex matcher engine class, implements reflex::PatternMatcher pattern matching interface with scan...">reflex::Matcher</a></code> only recognizes group captures at the top level of the regex (i.e. among the top-level alternations), because it uses an efficient FSM for matching.</p>
<p>By contrast, the PCRE2 matcher can capture groups within a regex:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="pcre2matcher_8h.html">reflex/pcre2matcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="comment">// a PCRE2Matcher to find &#39;TODO&#39; lines on stdin and capture their content to display</span></div><div class="line"><a class="code" href="classreflex_1_1_p_c_r_e2_matcher.html">reflex::PCRE2Matcher</a> matcher(<span class="stringliteral">&quot;TODO ([^\\n]+)&quot;</span>, stdin);</div><div class="line"><span class="keywordflow">while</span> (matcher.find())</div><div class="line">  std::cout</div><div class="line">    &lt;&lt; matcher.lineno() &lt;&lt; <span class="stringliteral">&quot;: &quot;</span></div><div class="line">    &lt;&lt; std::string(matcher[1].first, matcher[1].second)</div><div class="line">    &lt;&lt; std::endl;</div></div><!-- fragment --><p>The Boost.Regex library also supports group captures. It als supports partial matches, but that feature appears to be broken, so all input must be buffered:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="comment">// a BoostMatcher to find &#39;TODO&#39; lines on stdin and capture their content to display</span></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher(<span class="stringliteral">&quot;TODO ([^\\n]+)&quot;</span>, stdin);</div><div class="line">matcher.buffer(); <span class="comment">// because Boost.Regex partial_match is broken!</span></div><div class="line"><span class="keywordflow">while</span> (matcher.find())</div><div class="line">  std::cout</div><div class="line">    &lt;&lt; matcher.lineno() &lt;&lt; <span class="stringliteral">&quot;: &quot;</span></div><div class="line">    &lt;&lt; std::string(matcher[1].first, matcher[1].second)</div><div class="line">    &lt;&lt; std::endl;</div></div><!-- fragment --><p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 8</h3>
<p>This is a more advanced example, in which we will use the <code><a class="el" href="classreflex_1_1_boost_matcher.html" title="Boost matcher engine class implements reflex::PatternMatcher pattern matching interface with scan...">reflex::BoostMatcher</a></code> class to decompose URLs into parts: the host, port, path, optional ?-query string key=value pairs, and an optional #-anchor.</p>
<p>To do so, we change the pattern of the matcher to partially match each of the URL's parts and also use <code>input()</code> to check the input character:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span></div><div class="line"><span class="preprocessor">#include &lt;iostream&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacereflex.html">reflex</a>;</div><div class="line"></div><div class="line"><span class="keywordtype">int</span> <a class="code" href="reflex_8cpp.html#a3c04138a5bfe5d72780bb7e82a18e627">main</a>(<span class="keywordtype">int</span> argc, <span class="keywordtype">char</span> **argv)</div><div class="line">{</div><div class="line">  <span class="keywordflow">if</span> (argc &lt; 2)</div><div class="line">  {</div><div class="line">    std::cerr &lt;&lt; <span class="stringliteral">&quot;Usage: url &#39;URL&#39;&quot;</span> &lt;&lt; std::endl;</div><div class="line">    exit(EXIT_SUCCESS);</div><div class="line">  }</div><div class="line"></div><div class="line">  <a class="code" href="classreflex_1_1_boost_matcher.html">BoostMatcher</a> re(<span class="stringliteral">&quot;https?://([^:/]*):?(\\d*)/?([^?#]*)&quot;</span>, argv[1]);</div><div class="line"></div><div class="line">  <span class="keywordflow">if</span> (re.scan())</div><div class="line">  {</div><div class="line">    <span class="comment">// found a partial match at start, now check if we have a host</span></div><div class="line">    <span class="keywordflow">if</span> (re[1].first != NULL)</div><div class="line">    {</div><div class="line">      std::string host(re[1].first, re[1].second);</div><div class="line">      std::cout &lt;&lt; <span class="stringliteral">&quot;host: &quot;</span> &lt;&lt; host &lt;&lt; std::endl;</div><div class="line"></div><div class="line">      <span class="comment">// check of we have a port</span></div><div class="line">      <span class="keywordflow">if</span> (re[2].first != NULL &amp;&amp; re[2].second != 0)</div><div class="line">      {</div><div class="line">        std::string port(re[2].first, re[2].second);</div><div class="line">        std::cout &lt;&lt; <span class="stringliteral">&quot;port: &quot;</span> &lt;&lt; port &lt;&lt; std::endl;</div><div class="line">      }</div><div class="line"></div><div class="line">      <span class="comment">// check of we have a path</span></div><div class="line">      <span class="keywordflow">if</span> (re[3].first != NULL &amp;&amp; re[3].second != 0)</div><div class="line">      {</div><div class="line">        std::string path(re[3].first, re[3].second);</div><div class="line">        std::cout &lt;&lt; <span class="stringliteral">&quot;path: &quot;</span> &lt;&lt; path &lt;&lt; std::endl;</div><div class="line">      }</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="comment">// check if we have a query string</span></div><div class="line">    <span class="keywordflow">if</span> (re.input() == <span class="charliteral">&#39;?&#39;</span>)</div><div class="line">    {</div><div class="line">      <span class="comment">// now switch patterns to match the rest of the input</span></div><div class="line">      <span class="comment">// i.e. a query string or an anchor</span></div><div class="line"><span class="preprocessor">#if 0</span></div><div class="line">      <span class="comment">// 1st method: a pattern to split query strings at &#39;&amp;&#39;</span></div><div class="line">      re.pattern(<span class="stringliteral">&quot;&amp;&quot;</span>);</div><div class="line">      <span class="keywordflow">while</span> (re.split())</div><div class="line">        std::cout &lt;&lt; <span class="stringliteral">&quot;query: &quot;</span> &lt;&lt; re &lt;&lt; std::endl;</div><div class="line"><span class="preprocessor">#else</span></div><div class="line">      <span class="comment">// 2nd method: a pattern to capture key-value pairs between the &#39;&amp;&#39;</span></div><div class="line">      re.pattern(<span class="stringliteral">&quot;([^=&amp;]*)=?([^&amp;]*)&amp;?&quot;</span>);</div><div class="line">      <span class="keywordflow">while</span> (re.scan())</div><div class="line">        std::cout &lt;&lt;</div><div class="line">          <span class="stringliteral">&quot;query key: &quot;</span> &lt;&lt; std::string(re[1].first, re[1].second) &lt;&lt;</div><div class="line">          <span class="stringliteral">&quot;, value: &quot;</span> &lt;&lt; std::string(re[2].first, re[2].second) &lt;&lt; std::endl;</div><div class="line"><span class="preprocessor">#endif</span></div><div class="line">    }</div><div class="line">    <span class="keywordflow">else</span> <span class="keywordflow">if</span> (!re.at_end())</div><div class="line">    {</div><div class="line">      <span class="comment">// not a query string and not the end, we expect an # anchor</span></div><div class="line">      std::cout &lt;&lt; <span class="stringliteral">&quot;anchor: &quot;</span> &lt;&lt; re.rest() &lt;&lt; std::endl;</div><div class="line">    }</div><div class="line">  }</div><div class="line">  <span class="keywordflow">else</span></div><div class="line">  {</div><div class="line">    std::cout &lt;&lt; <span class="stringliteral">&quot;Error, not a http/s URL: &quot;</span> &lt;&lt; re.rest() &lt;&lt; std::endl;</div><div class="line">  }</div><div class="line"></div><div class="line">  <span class="keywordflow">return</span> EXIT_SUCCESS;</div><div class="line">}</div></div><!-- fragment --><p>Note that there are two ways to split the query string into key-value pairs. Both methods are shown in the two <code>#if</code> branches in the code above, with the first branch disabled with <code>#if 0</code>.</p>
<p>When executing </p><pre class="fragment">./url 'https://localhost:8080/test/me?name=reflex&amp;license=BSD-3'
</pre><p>this code prints: </p><pre class="fragment">host: localhost
port: 8080
path: test/me
query key: name, value: reflex
query key: license, value: BSD-3
</pre><p>🔝 <a href="#">Back to table of contents</a></p>
<h3>Example 9</h3>
<p>This example shows how a <code>FILE*</code> file descriptor is used as input. The file encoding is obtained from the UTF BOM, when present in the file. Note that the file's state is accessed through the matcher's member variable <code>in</code>:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="boostmatcher_8h.html">reflex/boostmatcher.h</a>&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">using namespace </span><a class="code" href="namespacereflex.html">reflex</a>;</div><div class="line"></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">BoostMatcher</a> matcher(<span class="stringliteral">&quot;\\s+&quot;</span>, fopen(<span class="stringliteral">&quot;filename&quot;</span>, <span class="stringliteral">&quot;r&quot;</span>));</div><div class="line"></div><div class="line"><span class="keywordflow">if</span> (matcher.in.file() != NULL &amp;&amp; matcher.in.good())</div><div class="line">{</div><div class="line">  <span class="keywordflow">switch</span> (matcher.in.file_encoding())</div><div class="line">  {</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#af213200d1cc11e672be420f1a4b61566">Input::file_encoding::plain</a>:      std::cout &lt;&lt; <span class="stringliteral">&quot;plain ASCII/binary/UTF-8&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a20bd3b19b413e61afc3c665da1c64d0e">Input::file_encoding::utf8</a>:       std::cout &lt;&lt; <span class="stringliteral">&quot;UTF-8 with BOM&quot;</span>;           <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a70bb4ad30b01c2b1b5061e105437d4de">Input::file_encoding::utf16be</a>:    std::cout &lt;&lt; <span class="stringliteral">&quot;UTF-16 big endian&quot;</span>;        <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#abe3c1db3aa8e7780eab85e201011f29e">Input::file_encoding::utf16le</a>:    std::cout &lt;&lt; <span class="stringliteral">&quot;UTF-16 little endian&quot;</span>;     <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a53577ddf6cf6f2d2f6bbf9630140273a">Input::file_encoding::utf32be</a>:    std::cout &lt;&lt; <span class="stringliteral">&quot;UTF-32 big endian&quot;</span>;        <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#af7ab91057731e548275a4eb63b6261ae">Input::file_encoding::utf32le</a>:    std::cout &lt;&lt; <span class="stringliteral">&quot;UTF-32 little endian&quot;</span>;     <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a70850a47577443173f3c31303612cdcf">Input::file_encoding::latin</a>:      std::cout &lt;&lt; <span class="stringliteral">&quot;ASCII+Latin-1/ISO-8859-1&quot;</span>; <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a533d3062b2eabd57d8d996f05a961c53">Input::file_encoding::cp437</a>:      std::cout &lt;&lt; <span class="stringliteral">&quot;DOS CP 437&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a2c7cd01f85b15b2792a04021c415805d">Input::file_encoding::cp850</a>:      std::cout &lt;&lt; <span class="stringliteral">&quot;DOS CP 850&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#aec2e4fc22c907af11338e829794c243f">Input::file_encoding::cp858</a>:      std::cout &lt;&lt; <span class="stringliteral">&quot;DOS CP 858&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#ad80ba65b2eb9e237939b29a4758d8dda">Input::file_encoding::ebcdic</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;EBCDIC&quot;</span>;                   <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a1ecaa0c4f552a531c0eccfbe5ab006f1">Input::file_encoding::cp1250</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;Windows CP 1250&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#ac463724c06802db189b6aeb3b2ce0c04">Input::file_encoding::cp1251</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;Windows CP 1251&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a3ba20632cdb12411e69d29ac450037a3">Input::file_encoding::cp1252</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;Windows CP 1252&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#ac7ff7de99f3952909f66633f36c1fd25">Input::file_encoding::cp1253</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;Windows CP 1253&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#ac194ff983cbb3daafc103f52d220a912">Input::file_encoding::cp1254</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;Windows CP 1254&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#af559c1cfe8f5f60572c6ae4b715ec146">Input::file_encoding::cp1255</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;Windows CP 1255&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a8f7bd312945679f2d67dd0ea5eab44bf">Input::file_encoding::cp1256</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;Windows CP 1256&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#ab18e5f97380df73e18ff084987e41766">Input::file_encoding::cp1257</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;Windows CP 1257&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a3a978fa6caf666fd367adee792d2b799">Input::file_encoding::cp1258</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;Windows CP 1258&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a8a81b45fbf259ec1d13935375c35481d">Input::file_encoding::iso8859_2</a>:  std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-2&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a6ac76fc3cd8539231e452a80a02e631d">Input::file_encoding::iso8859_3</a>:  std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-3&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a27a8b9e0e2cc441036d4b4a926485e8a">Input::file_encoding::iso8859_4</a>:  std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-4&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a53ed11fde3562f4d5c53581bbe2d7608">Input::file_encoding::iso8859_5</a>:  std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-5&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a0f5cba1a52a8e5d484e0613c730be931">Input::file_encoding::iso8859_6</a>:  std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-6&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#aaf9516e1d489edd6b7aae831b0d27856">Input::file_encoding::iso8859_7</a>:  std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-7&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a5035cf2b775ab05446b61c84558697c5">Input::file_encoding::iso8859_8</a>:  std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-8&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a90d5c3ee945f132789a2805656547938">Input::file_encoding::iso8859_9</a>:  std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-9&quot;</span>;               <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#ac8d51f0373dbf4e65c404e6e08cc4ce9">Input::file_encoding::iso8859_10</a>: std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-10&quot;</span>;              <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a885b87ce71146c49f8eceef4d939cef5">Input::file_encoding::iso8859_11</a>: std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-11&quot;</span>;              <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a56a15bc6091f84a47083719c1ac84f4b">Input::file_encoding::iso8859_13</a>: std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-13&quot;</span>;              <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a2633104718050d152b044160e79b1396">Input::file_encoding::iso8859_14</a>: std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-14&quot;</span>;              <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a446455dea74b6e8995036ff3cc2dab82">Input::file_encoding::iso8859_15</a>: std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-15&quot;</span>;              <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a96ecc25ba97a877c5d03ec3f32b44464">Input::file_encoding::iso8859_16</a>: std::cout &lt;&lt; <span class="stringliteral">&quot;ISO-8859-16&quot;</span>;              <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> Input::file_encoding::mac_roman:  std::cout &lt;&lt; <span class="stringliteral">&quot;Macintosh Roman&quot;</span>;          <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a8f1795ff2b9505ff8bc114721f40b336">Input::file_encoding::koi8_r</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;KOI8-R&quot;</span>;                   <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#afe9401d703dd4e82d63e694118c2bf39">Input::file_encoding::koi8_u</a>:     std::cout &lt;&lt; <span class="stringliteral">&quot;KOI8-U&quot;</span>;                   <span class="keywordflow">break</span>;</div><div class="line">    <span class="keywordflow">case</span> <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#ab7453c75ea45732498a8cbd18e38aa7c">Input::file_encoding::koi8_ru</a>:    std::cout &lt;&lt; <span class="stringliteral">&quot;KOI8-RU&quot;</span>;                  <span class="keywordflow">break</span>;</div><div class="line">  }</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot; of &quot;</span> &lt;&lt; matcher.in.size() &lt;&lt; <span class="stringliteral">&quot; converted bytes to read\n&quot;</span>;</div><div class="line">  matcher.buffer(); <span class="comment">// because Boost.Regex partial_match is broken!</span></div><div class="line">  <span class="keywordflow">if</span> (matcher.split() != 0)</div><div class="line">    std::cout &lt;&lt; <span class="stringliteral">&quot;Starts with: &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Rest of the file is: &quot;</span> &lt;&lt; matcher.rest();</div><div class="line">  fclose(matcher.in.file());</div><div class="line">}</div></div><!-- fragment --><p>The default encoding is <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#af213200d1cc11e672be420f1a4b61566" title="plain octets: 7-bit ASCII, 8-bit binary or UTF-8 without BOM detected ">reflex::Input::file_encoding::plain</a></code> when no UTF BOM is detected at the start of the input file. The encodings <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a70850a47577443173f3c31303612cdcf" title="ISO-8859-1, Latin-1. ">reflex::Input::file_encoding::latin</a></code>, <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a3ba20632cdb12411e69d29ac450037a3" title="Windows CP 1252. ">reflex::Input::file_encoding::cp1252</a></code>, <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a533d3062b2eabd57d8d996f05a961c53" title="DOS CP 437. ">reflex::Input::file_encoding::cp437</a></code>, <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a2c7cd01f85b15b2792a04021c415805d" title="DOS CP 850. ">reflex::Input::file_encoding::cp850</a></code>, <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#ad80ba65b2eb9e237939b29a4758d8dda" title="EBCDIC. ">reflex::Input::file_encoding::ebcdic</a></code> are never detected automatically, because plain encoding is implicitly assumed to be the default encoding. To convert these files, set the file encoding format explicitly in your code. For example, if you expect the source file to contain ISO-8859-1 8-bit characters (ASCII and the latin-1 supplement) then set the default file encoding to <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a70850a47577443173f3c31303612cdcf" title="ISO-8859-1, Latin-1. ">reflex::Input::file_encoding::latin</a></code> as follows:</p>
<div class="fragment"><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> input(fopen(<span class="stringliteral">&quot;filename&quot;</span>, <span class="stringliteral">&quot;r&quot;</span>), <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a70850a47577443173f3c31303612cdcf">reflex::Input::file_encoding::latin</a>);</div><div class="line"><span class="keywordflow">if</span> (input.<a class="code" href="classreflex_1_1_input.html#a9f64e5c813a08fd7a65819074ffcd20a">file</a>() != NULL)</div><div class="line">{</div><div class="line">  <a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher(<span class="stringliteral">&quot;\\s+&quot;</span>, input);</div><div class="line">  matcher.buffer(); <span class="comment">// because Boost.Regex partial_match is broken!</span></div><div class="line">  <span class="keywordflow">if</span> (matcher.split() != 0)</div><div class="line">    std::cout &lt;&lt; <span class="stringliteral">&quot;Starts with: &quot;</span> &lt;&lt; matcher.text() &lt;&lt; std::endl;</div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;Rest of the file is: &quot;</span> &lt;&lt; matcher.rest();</div><div class="line">  fclose(input.<a class="code" href="classreflex_1_1_input.html#a9f64e5c813a08fd7a65819074ffcd20a">file</a>());</div><div class="line">}</div></div><!-- fragment --><p>This sets the file encoding to ISO-8859-1, but only if no UTF BOM was detected in the file. Files with a UTF BOM are always decoded as UTF, which cannot be overruled.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h1><a class="anchor" id="tricks"></a>
Tips, tricks, and gotchas                                             </h1>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="extern-yyin"></a>
Errors when declaring extern yyin, yytext, yylineno              </h2>
<p>For backward compatibility with Flex, option <code>−−flex</code> defines macros to expand <code>yyin</code>, <code>yyout</code>, <code>yylineno</code>, <code>yytext</code>, and <code>yyleng</code>. The macro expansion depends on the <code>−−bison</code> option or <code>−−bison-locations</code>, <code>−−bison-cc</code> and so on.</p>
<p>When used with <code>−−flex</code>, option <code>−−bison</code> generates global "yy" variables and functions, see <a class="el" href="index.html#reflex-bison-mt-safe">Bison and thread-safety</a> for details. This means that <code>yytext</code>, <code>yyleng</code>, and <code>yylineno</code> are global variables. More specifically, the following declarations are generated with <code>−−flex</code> and <code>−−bison</code>:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="keywordtype">char</span> *<a class="code" href="flexlexer_8h.html#a0d71f919dbec1ffd74b2460fa7e5ac28">yytext</a>;</div><div class="line"><a class="code" href="flexlexer_8h.html#aa9522b9e313cd74e4e688e358cbca836">yy_size_t</a> <a class="code" href="flexlexer_8h.html#afa07a629486cb790560bb95713ec7794">yyleng</a>;</div><div class="line"><span class="keywordtype">int</span> <a class="code" href="flexlexer_8h.html#ad71cf0fddcfe4f61de0929105b33226c">yylineno</a>;</div><div class="line"><span class="keywordtype">int</span> yylex();</div></div><!-- fragment --> </div><p>Note that <code>yyin</code> is not a global variable, because the <code>yyin</code> macro expands to a pointer to the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> of the matcher. This offers advanced input handling capabilities that may be useful.</p>
<p>However, the following declaration, when present in a Lex/Flex lexer specification, causes a compilation error:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="keyword">extern</span> FILE *<a class="code" href="flexlexer_8h.html#aa789f4617e33fb99594cb04a3688a0c1">yyin</a>; <span class="comment">// ERROR: fails to compile (remove this line)</span></div></div><!-- fragment --> </div><p>Option <code>−−yy</code> enables <code>−−flex</code> and <code>−−bison</code>. In addition, this option generates the following declarations to define the <code>yyin</code> and <code>yyout</code> as global <code>FILE*</code> type variables:</p>
<div class="alt"> <div class="fragment"><div class="line">FILE *<a class="code" href="flexlexer_8h.html#aa789f4617e33fb99594cb04a3688a0c1">yyin</a>;</div><div class="line">FILE *<a class="code" href="flexlexer_8h.html#a4fd44867d448dcb6fc32ea004a15de54">yyout</a>;</div></div><!-- fragment --> </div><p>Note that without option <code>−−yy</code>, when options <code>−−flex</code> and <code>−−bison</code> are used, <code>yyin</code> is a pointer to a <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> object. This means that <code>yyin</code> is not restricted to <code>FILE*</code> types:</p>
<div class="alt"> <div class="fragment"><div class="line">std::ifstream ifs(filename, std::ios::in);</div><div class="line">yyin = &amp;ifs; <span class="comment">// FILE* or std::istream</span></div></div><!-- fragment --> </div><div class="alt"> <div class="fragment"><div class="line">yyin = <span class="stringliteral">&quot;...&quot;</span>; <span class="comment">// (wide) strings</span></div></div><!-- fragment --> </div><p>See <a class="el" href="index.html#reflex-input">Switching input sources </a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="invalid-utf"></a>
Invalid UTF encodings                                            </h2>
<p>It may be tempting to write a pattern with <code>.</code> (dot) as a wildcard in a lexer specification, but beware that in Unicode mode enabled with <em><code>%option unicode</code></em> or with modifier <code>(?u:φ)</code>, the dot matches any code point, including code points outside of the valid Unicode character range and invalid overlong UTF-8 (except that it won't match newline unless <em><code>%option dotall</code></em> is specified.) The reason for this design choice is that a lexer should be able to implement a "catch all else" rule to report errors in the input:</p>
<div class="alt"> <div class="fragment"><div class="line">.    std::cerr &lt;&lt; <span class="stringliteral">&quot;lexical error, full stop!&quot;</span> &lt;&lt; std::endl;</div><div class="line">     <span class="keywordflow">return</span> 0;</div></div><!-- fragment --> </div><p>If dot in Unicode mode would be restricted to match valid Unicode only, then the action above will never be triggered when invalid input is encountered. Because all non-dot regex patterns are valid Unicode in RE/flex, it would be impossible to write a "catch all else" rule that catches input format errors!</p>
<p>The dot in Unicode mode is self-synchronizing and consumes text up to to the next ASCII or Unicode character.</p>
<p>To accept valid Unicode input in regex patterns, make sure to avoid <code>.</code> (dot) and use <code>\p{Unicode}</code> or <code>\X</code> instead, and reserve dot to catch anything, such as invalid UTF encodings. We use <code>.|\n</code> or <em><code>%option dotall</code></em> to catch anything including <code>\n</code> and invalid UTF-8/16/32 encodings.</p>
<p>Furthermore, before matching any input, invalid UTF-16 input is detected automatically by the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class and replaced with the <code><a class="el" href="utf8_8h.html#a75f9fce49b2f30fb97235c76cd70d32f" title="Replace invalid UTF-8 with the non-character U+200000 code point for guaranteed error detection (the ...">REFLEX_NONCHAR</a></code> code point U+200000 that lies outside the valid Unicode range. This code point is never matched by non-dot regex patterns and is easy to detect by a regex pattern with a dot and a corresponding error action as shown above.</p>
<p>Note that character classes written as bracket lists may produce invalid Unicode ranges when used improperly. This is not a problem for matching, but may prevent rejecting surrogate halves that are invalid Unicode. For example, <code>[\u{00}-\u{10FFFF}]</code> obviously includes the invalid range of surrogate halves <code>[\u{D800}-\u{DFFF}]</code>. You can always remove surrogate halves from any character class by intersecting the class with <code>[\p{Unicode}]</code>, that is <code>[...&amp;&amp;[\p{Unicode}]]</code>. Furthermore, character class negation with <code>^</code> results in classes that are within range U+0000 to U+10FFFF and excludes surrogate halves.</p>
<p>🔝 <a href="#">Back to contents</a></p>
<h2><a class="anchor" id="errors"></a>
Error reporting and recovery                                          </h2>
<p>When your scanner or parser encounters an error in the input, the scanner or parser should report it and attempt to continue processing the input by recovering from the error condition. Most compilers recover from an error to continue processing the input until a threshold on the maximum number of errors is exceeded.</p>
<p>In our lexer specification of a scanner, we may define a "catch all else" rule with pattern <code>.</code> to report an unmatched "mystery character" that is not recognized, for example:</p>
<div class="alt"> <div class="fragment"><div class="line">%<span class="keyword">class</span>{</div><div class="line">  <span class="keyword">static</span> <span class="keyword">const</span> <span class="keywordtype">size_t</span> max_errors = 10;</div><div class="line">  <span class="keywordtype">size_t</span> errors;</div><div class="line">}</div><div class="line"></div><div class="line">%init{</div><div class="line">  errors = 0;</div><div class="line">}</div><div class="line"></div><div class="line">%%</div><div class="line">...  <span class="comment">// lexer rules</span></div><div class="line"></div><div class="line">.    std::string line = matcher().line();</div><div class="line">     std::cerr &lt;&lt; <span class="stringliteral">&quot;Error: mystery character at line &quot;</span> &lt;&lt; lineno() &lt;&lt; <span class="stringliteral">&quot;:\n&quot;</span> &lt;&lt; line &lt;&lt; std::endl;</div><div class="line">     <span class="keywordflow">for</span> (<span class="keywordtype">size_t</span> i = columno(); i &gt; 0; --i)</div><div class="line">       std::cerr &lt;&lt; <span class="stringliteral">&quot; &quot;</span>;</div><div class="line">     std::cerr &lt;&lt; <span class="stringliteral">&quot;\\__ here&quot;</span> &lt;&lt; std::endl;</div><div class="line">     <span class="keywordflow">if</span> (++errors &gt;= max_errors)</div><div class="line">       <span class="keywordflow">return</span> 0;</div><div class="line">%%</div></div><!-- fragment --> </div><p>The error message indicates the offending line number with <code>lineno()</code> and prints the problematic line of input using <code>matcher().line()</code>. The position on the line is indicated with an arrow placed below the line at offset <code>columno()</code> from the start of the line, where <code>columno()</code> takes tabs and wide characters into account.</p>
<p>This error message does not take the window width into account, which may result in misplacing the arrow when the line is too long and overflows onto the next rows in the window, unless changes are made to the code to print the relevant part of the line only.</p>
<p>There are other ways to indicate the location of an error, for example as <code>--&gt;</code> <code>&lt;--</code> and highlighting the error using the ANSI SGI escape sequence for bold typeface:</p>
<div class="alt"> <div class="fragment"><div class="line">.    std::string line = lexer-&gt;matcher().line();</div><div class="line">     std::string before = line.substr(0, border());</div><div class="line">     std::string after = line.substr(border() + size());</div><div class="line">     std::cerr &lt;&lt; <span class="stringliteral">&quot;Error: mystery character at line &quot;</span> &lt;&lt; lineno() &lt;&lt; <span class="stringliteral">&quot;:&quot;</span> &lt;&lt; std::endl;</div><div class="line">     std::cerr &lt;&lt; before &lt;&lt; <span class="stringliteral">&quot;\033[1m --&gt; &quot;</span> &lt;&lt; str() &lt;&lt; <span class="stringliteral">&quot; &lt;-- \033[0m&quot;</span> &lt;&lt; after &lt;&lt; std::endl;</div><div class="line">     <span class="keywordflow">if</span> (++errors &gt;= max_errors)</div><div class="line">       <span class="keywordflow">return</span> 0;</div></div><!-- fragment --> </div><p>This prints the start of the line up to the mismatching position on the line returned by <code>border()</code>, followed by the highlighted "mystery character". Beware that this can be a control code or invalid Unicode code point, which should be checked before displaying it.</p>
<p>This scanner terminates when 10 lexical errors are encountered in the input, as defined by <code>max_errors</code>.</p>
<p>By default, Bison invokes <code>yyerror()</code> (or <code>yy::parser::error()</code> with <a class="el" href="index.html#reflex-bison-cc">Bison-cc</a> parsers) to report syntax errors. However, it is recommended to use Bison error productions to handle and resolve syntax errors intelligently by synchronizing on tokens that allow the parser to continue, for example on a semicolon in a <a class="el" href="index.html#reflex-bison-bridge">Bison-bridge</a> parser:</p>
<div class="alt"> <div class="fragment"><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span></div><div class="line"><span class="preprocessor">  #define YYPARSE_PARAM lexer</span></div><div class="line"><span class="preprocessor">  #define YYLEX_PARAM   lexer</span></div><div class="line">  <span class="keywordtype">void</span> yyerror(Lexer *lexer, <span class="keyword">const</span> <span class="keywordtype">char</span> *msg);</div><div class="line">%}</div><div class="line"></div><div class="line">%pure-parser</div><div class="line">%lex-param { Lexer *lexer }</div><div class="line">%parse-param { Lexer *lexer }</div><div class="line"></div><div class="line">%%</div><div class="line">...          <span class="comment">// grammar rules</span></div><div class="line"></div><div class="line">| error <span class="charliteral">&#39;;&#39;</span>  { yyerrok; <span class="keywordflow">if</span> (++lexer-&gt;errors &gt;= lexer-&gt;max_errors) YYABORT; }</div><div class="line">;</div><div class="line">%%</div></div><!-- fragment --> </div><p>Note that the lexer keeps track of the number of errors. When the maximum number of lexical and syntax errors is reached, we bail out.</p>
<p>The line of input where the syntax error occurs is reported with <code>yyerror()</code> for the <a class="el" href="index.html#reflex-bison-bridge">Bison-bridge</a> parser:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="keywordtype">void</span> yyerror(Lexer *lexer, <span class="keyword">const</span> <span class="keywordtype">char</span> *msg)</div><div class="line">{</div><div class="line">  std::string line = lexer-&gt;matcher().line();</div><div class="line">  std::string before = line.substr(0, border());</div><div class="line">  std::string after = line.substr(border() + size());</div><div class="line">  std::cerr &lt;&lt; <span class="stringliteral">&quot;Error: &quot;</span> &lt;&lt; msg &lt;&lt; <span class="stringliteral">&quot; at line &quot;</span> &lt;&lt; lexer-&gt;lineno() &lt;&lt; <span class="stringliteral">&quot;:&quot;</span> &lt;&lt; std::endl;</div><div class="line">  std::cerr &lt;&lt; before &lt;&lt; <span class="stringliteral">&quot;\033[1m --&gt; &quot;</span> &lt;&lt; lexer-&gt;str() &lt;&lt; <span class="stringliteral">&quot; &lt;-- \033[0m&quot;</span> &lt;&lt; after &lt;&lt; std::endl;</div><div class="line">}</div></div><!-- fragment --> </div><p>With option <code>−−flex</code>, the definitions part of the lexer specification is updated as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">%{</div><div class="line"><span class="preprocessor">  #include &quot;lex.yy.h&quot;</span></div><div class="line">  <span class="keywordtype">void</span> yyerror(yyscan_t, <span class="keyword">const</span> <span class="keywordtype">char</span>*);</div><div class="line"><span class="preprocessor">  #define YYPARSE_PARAM scanner</span></div><div class="line"><span class="preprocessor">  #define YYLEX_PARAM   scanner</span></div><div class="line">%}</div><div class="line"></div><div class="line">%option flex</div><div class="line"></div><div class="line">%pure-parser</div><div class="line">%lex-param { <span class="keywordtype">void</span> *scanner }</div><div class="line">%parse-param { <span class="keywordtype">void</span> *scanner }</div></div><!-- fragment --> </div><p>And the <code>yyerror()</code> function is updated as follows:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="keywordtype">void</span> yyerror(yyscan_t scanner, <span class="keyword">const</span> <span class="keywordtype">char</span> *msg)</div><div class="line">{</div><div class="line">  yyFlexLexer *lexer = <span class="keyword">static_cast&lt;</span>yyscanner_t*<span class="keyword">&gt;</span>(scanner);</div><div class="line">  std::string line = lexer-&gt;matcher().line();</div><div class="line">  std::string before = line.substr(0, border());</div><div class="line">  std::string after = line.substr(border() + size());</div><div class="line">  std::cerr &lt;&lt; <span class="stringliteral">&quot;Error: &quot;</span> &lt;&lt; msg &lt;&lt; <span class="stringliteral">&quot; at line &quot;</span> &lt;&lt; lexer-&gt;lineno() &lt;&lt; <span class="stringliteral">&quot;:&quot;</span> &lt;&lt; std::endl;</div><div class="line">  std::cerr &lt;&lt; before &lt;&lt; <span class="stringliteral">&quot;\033[1m --&gt; &quot;</span> &lt;&lt; lexer-&gt;str() &lt;&lt; <span class="stringliteral">&quot; &lt;-- \033[0m&quot;</span> &lt;&lt; after &lt;&lt; std::endl;</div><div class="line">}</div></div><!-- fragment --> </div><p>These examples assume that the syntax error was detected immediately at the last token scanned and displayed with <code>lexer-&gt;str()</code>, which may not always be the case.</p>
<p>With <a class="el" href="index.html#reflex-bison-bridge-locations">Bison-bridge &amp; locations</a> parsers (and optionally <code>−−flex</code>), we obtain the first and the last line of an error and we can use this information to report the error. For example as follows:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="keywordtype">void</span> yyerror(YYLTYPE *yylloc, yyscan_t scanner, <span class="keyword">const</span> <span class="keywordtype">char</span> *msg)</div><div class="line">{</div><div class="line">  yyFlexLexer *lexer = <span class="keyword">static_cast&lt;</span>yyscanner_t*<span class="keyword">&gt;</span>(scanner);</div><div class="line">  std::cerr &lt;&lt; <span class="stringliteral">&quot;Error: &quot;</span> &lt;&lt; msg &lt;&lt; <span class="stringliteral">&quot; at line &quot;</span> &lt;&lt; yylloc-&gt;first_line &lt;&lt; <span class="stringliteral">&quot;:&quot;</span> &lt;&lt; std::endl;</div><div class="line">  <span class="keywordflow">if</span> (yylloc-&gt;first_line == yylloc-&gt;last_line &amp;&amp; yylloc-&gt;first_line == lexer-&gt;lineno())</div><div class="line">  {</div><div class="line">    std::cerr &lt;&lt; lexer-&gt;matcher().line() &lt;&lt; std::endl;</div><div class="line">    <span class="keywordflow">for</span> (<span class="keywordtype">int</span> i = 0; i &lt; yylloc-&gt;first_column; ++i)</div><div class="line">       std::cerr &lt;&lt; <span class="stringliteral">&quot; &quot;</span>;</div><div class="line">    <span class="keywordflow">for</span> (<span class="keywordtype">int</span> i = yylloc-&gt;first_column; i &lt;= yylloc-&gt;last_column; ++i)</div><div class="line">       std::cerr &lt;&lt; <span class="stringliteral">&quot;~&quot;</span>;</div><div class="line">    std::cerr &lt;&lt; std::endl;</div><div class="line">  }</div><div class="line">  <span class="keywordflow">else</span></div><div class="line">  {</div><div class="line">    FILE *file = lexer-&gt;in().file(); <span class="comment">// the current FILE* being scanned</span></div><div class="line">    <span class="keywordflow">if</span> (file != NULL)</div><div class="line">    {</div><div class="line">      <a class="code" href="flexlexer_8h.html#af8ef002da9f9ba35951176ec0be7ab23">YY_BUFFER_STATE</a> buf = <a class="code" href="flexlexer_8h.html#aa28e58c21d4fbe7132b6de8edf76e43a">yy_create_buffer</a>(file, <a class="code" href="flexlexer_8h.html#ae7e51116e747d3390e7a6cfc6532834c">YY_BUF_SIZE</a>, scanner);</div><div class="line">      <a class="code" href="flexlexer_8h.html#a3807e0a231906457045e429d6721f291">yypush_buffer_state</a>(buf, scanner); <span class="comment">// push current buffer (matcher), use buf</span></div><div class="line">      off_t pos = ftell(file); <span class="comment">// save current position in the file</span></div><div class="line">      fseek(file, 0, SEEK_SET); <span class="comment">// go to the start of the file</span></div><div class="line">      <span class="keywordflow">for</span> (<span class="keywordtype">int</span> i = 1; i &lt; yylloc-&gt;first_line; ++i)</div><div class="line">        buf-&gt;skip(<span class="charliteral">&#39;\n&#39;</span>); <span class="comment">// skip to the next line</span></div><div class="line">      <span class="keywordflow">for</span> (<span class="keywordtype">int</span> i = yylloc-&gt;first_line; i &lt;= yylloc-&gt;last_line; ++i)</div><div class="line">      {</div><div class="line">        std::cerr &lt;&lt; buf-&gt;line() &lt;&lt; std::endl; <span class="comment">// display offending line</span></div><div class="line">        buf-&gt;skip(<span class="charliteral">&#39;\n&#39;</span>); <span class="comment">// next line</span></div><div class="line">      }</div><div class="line">      fseek(file, pos, SEEK_SET); <span class="comment">// restore position in the file to continue scanning</span></div><div class="line">      <a class="code" href="flexlexer_8h.html#ad0d14a86a4c2feaa7e9e73fd26239dc1">yypop_buffer_state</a>(scanner); <span class="comment">// restore buffer (matcher)</span></div><div class="line">    }</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --> </div><p>Because we use Flex-compatible reentrant functions <code><a class="el" href="flexlexer_8h.html#aa28e58c21d4fbe7132b6de8edf76e43a" title="Flex-compatible macro: create and return a new buffer. ">yy_create_buffer()</a></code>, <code><a class="el" href="flexlexer_8h.html#a3807e0a231906457045e429d6721f291" title="Flex-compatible macro: push the current buffer on the stack to use the given buffer. ">yypush_buffer_state()</a></code>, and <code><a class="el" href="flexlexer_8h.html#ad0d14a86a4c2feaa7e9e73fd26239dc1" title="Flex-compatible macro: pop buffer from the stack and delete the current buffer. ">yypop_buffer_state()</a></code> that take an extra scanner argument, we also use options <code>−−flex</code> and <code>−−reentrant</code> in addition to <code>−−bison-bridge</code> and <code>−−bison-locations</code> to generate the reentrant scanner for the example shown above.</p>
<p>Similarly, with <a class="el" href="index.html#reflex-bison-complete-locations">Bison-complete &amp; locations</a> parsers, syntax errors can be reported as follows (without option <code>−−flex</code>):</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="keywordtype">void</span> yy::parser::error(<span class="keyword">const</span> location&amp; loc, <span class="keyword">const</span> std::string&amp; msg)</div><div class="line">{</div><div class="line">  std::cerr &lt;&lt; loc &lt;&lt; <span class="stringliteral">&quot;: &quot;</span> &lt;&lt; msg &lt;&lt; std::endl;</div><div class="line">  <span class="keywordflow">if</span> (loc.begin.line == loc.end.line &amp;&amp; loc.begin.line == lexer.lineno())</div><div class="line">  {</div><div class="line">    std::cerr &lt;&lt; lexer.matcher().line() &lt;&lt; std::endl;</div><div class="line">    <span class="keywordflow">for</span> (<span class="keywordtype">size_t</span> i = 0; i &lt; loc.begin.column; ++i)</div><div class="line">      std::cerr &lt;&lt; <span class="stringliteral">&quot; &quot;</span>;</div><div class="line">    <span class="keywordflow">for</span> (<span class="keywordtype">size_t</span> i = loc.begin.column; i &lt;= loc.end.column; ++i)</div><div class="line">      std::cerr &lt;&lt; <span class="stringliteral">&quot;~&quot;</span>;</div><div class="line">    std::cerr &lt;&lt; std::endl;</div><div class="line">  }</div><div class="line">  <span class="keywordflow">else</span></div><div class="line">  {</div><div class="line">    FILE *file = lexer.in().file(); <span class="comment">// the current file being scanned</span></div><div class="line">    <span class="keywordflow">if</span> (file != NULL)</div><div class="line">    {</div><div class="line">      yy::scanner::Matcher *m = lexer.new_matcher(file); <span class="comment">// new matcher</span></div><div class="line">      lexer.push_matcher(m); <span class="comment">// save the current matcher</span></div><div class="line">      off_t pos = ftell(file); <span class="comment">// save current position in the file</span></div><div class="line">      fseek(file, 0, SEEK_SET); <span class="comment">// go to the start of the file</span></div><div class="line">      <span class="keywordflow">for</span> (<span class="keywordtype">size_t</span> i = 1; i &lt; loc.begin.line; ++i)</div><div class="line">        m-&gt;skip(<span class="charliteral">&#39;\n&#39;</span>); <span class="comment">// skip to the next line</span></div><div class="line">      <span class="keywordflow">for</span> (<span class="keywordtype">size_t</span> i = loc.begin.line; i &lt;= loc.end.line; ++i)</div><div class="line">      {</div><div class="line">        std::cerr &lt;&lt; m-&gt;line() &lt;&lt; std::endl; <span class="comment">// display offending line</span></div><div class="line">        m-&gt;skip(<span class="charliteral">&#39;\n&#39;</span>); <span class="comment">// next line</span></div><div class="line">      }</div><div class="line">      fseek(file, pos, SEEK_SET); <span class="comment">// restore position in the file to continue scanning</span></div><div class="line">      lexer.pop_matcher(); <span class="comment">// restore matcher</span></div><div class="line">    }</div><div class="line">  }</div><div class="line">}</div></div><!-- fragment --> </div><p>If option <code>−−exception</code> is specified with a lexer specification, for example as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">%option exception=<span class="stringliteral">&quot;yy::parser::error(location(), \&quot;Unknown token.\&quot;)&quot;</span></div></div><!-- fragment --> </div><p>then we should make sure to consume some input in the exception handler to advance the scanner forward to skip the offending input and to allow the scanner to recover:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="keywordtype">void</span> yy::parser::error(<span class="keyword">const</span> location&amp; loc, <span class="keyword">const</span> std::string&amp; msg)</div><div class="line">{</div><div class="line">  <span class="keywordflow">if</span> (lexer.size() == 0) <span class="comment">// if token is unknown (no match)</span></div><div class="line">      lexer.matcher().winput(); <span class="comment">// skip character</span></div><div class="line">  ...</div><div class="line">}</div></div><!-- fragment --> </div><p>Error reporting can be combined with Bison Lookahead Correction (LAC), which is enabled with:</p>
<div class="alt"> <div class="fragment"><div class="line">%define parse.lac full</div></div><!-- fragment --> </div><p>For more details on Bison error messaging, resolution, and LAC, please see the Bison documentation.</p>
<p>🔝 <a href="#">Back to contents</a></p>
<h2><a class="anchor" id="setlocale"></a>
On using setlocale                                                 </h2>
<p>The RE/flex scanners and regex matchers use an internal buffer with UTF-8 encoded text content to scan wide strings and UTF-16/UTF-32 input. This means that Unicode input is normalized to UTF-8 prior to matching. This internal conversion is independent of the current C locale and is performed automatically by the <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> class that passes the UTF-8-normalized input to the matchers.</p>
<p>Furthermore, RE/flex lexers may invoke the <code>wstr()</code>, <code>wchr()</code>, and <code>wpair()</code> methods to extract wide string and wide character matches. These methods are also independent of the current C locale.</p>
<p>This means that setting the C locale in an application will not affect the performance of RE/flex scanners and regex matchers.</p>
<p>As a side note, to display wide strings properly and to save wide strings to UTF-8 text files, it is generally recommended to set the UTF-8 locale. For example:</p>
<div class="fragment"><div class="line">std::setlocale(LC_ALL, <span class="stringliteral">&quot;en_US.UTF-8&quot;</span>);        <span class="comment">// setlocale UTF-8</span></div><div class="line">std::ifstream ifs(<span class="stringliteral">&quot;file.txt&quot;</span>, std::ios::in);  <span class="comment">// open UTF-8/16/32 text file</span></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, ifs);    <span class="comment">// not affected by setlocale</span></div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">{</div><div class="line">  std::wstring&amp; match = matcher.wstr();       <span class="comment">// not affected by setlocale</span></div><div class="line">  std::wcout &lt;&lt; match &lt;&lt; std::endl;           <span class="comment">// affected by setlocale</span></div><div class="line">}</div><div class="line">ifs.close();</div></div><!-- fragment --><p>This displays wide string matches in UTF-8 on most consoles and terminals, but not on all systems (I'm looking at you, Mac OS X terminal!) Instead of <code>std::wcout</code> we can use <code>std::cout</code> instead to display UTF-8 content directly:</p>
<div class="fragment"><div class="line">std::setlocale(LC_ALL, <span class="stringliteral">&quot;en_US.UTF-8&quot;</span>);       <span class="comment">// setlocale UTF-8</span></div><div class="line">std::ifstream ifs(<span class="stringliteral">&quot;file.txt&quot;</span>, std::ios::in); <span class="comment">// open UTF-8/16/32 text file</span></div><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, ifs);   <span class="comment">// not affected by setlocale</span></div><div class="line"><span class="keywordflow">while</span> (matcher.find() != 0)</div><div class="line">{</div><div class="line">  std::string&amp; match = matcher.str();        <span class="comment">// not affected by setlocale</span></div><div class="line">  std::cout &lt;&lt; match &lt;&lt; std::endl;           <span class="comment">// not affected by setlocale</span></div><div class="line">}</div><div class="line">ifs.close();</div></div><!-- fragment --><p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="iso-8859-1"></a>
Scanning ISO-8859-1 (latin-1) files with a Unicode scanner        </h2>
<p>Scanning files encoded in ISO-8859-1 by a Unicode scanner that expects UTF-8 will cause the scanner to misbehave or throw errors.</p>
<p>Many text files are still encoded in ISO-8859-1 (also called latin-1). To set up your scanner to safely scan ISO-8859-1 content when your scanner rules use Unicode (with the <code>−−unicode</code> option and your patterns that use UTF-8 encodings), set the default file encoding to <code>latin</code>:</p>
<div class="fragment"><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> input(stdin, <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a70850a47577443173f3c31303612cdcf">reflex::Input::file_encoding::latin</a>);</div><div class="line">Lexer lexer(input);</div><div class="line">lexer.lex();</div></div><!-- fragment --><p>This scans files from standard input that are encoded in ISO-8859-1, unless the file has a <a href="www.unicode.org/faq/utf_bom.html">UTF Byte Order Mark (BOM)</a>. When a BOM is detected the scanner switches to UTF scanning.</p>
<p>See <a class="el" href="index.html#regex-input-file">FILE encodings</a> to set file encodings.</p>
<p>🔝 <a href="#">Back to contents</a></p>
<h2><a class="anchor" id="crlf"></a>
Handling DOS CRLF newlines                                              </h2>
<p>DOS files and other DOS or Windows input sources typically end lines with CRLF byte pairs. There are two ways to deal with CRLF pairs:</p>
<ol type="1">
<li>Use <code><a class="el" href="classreflex_1_1_input_1_1dos__streambuf.html" title="Stream buffer for reflex::Input to read DOS files, replaces CRLF by LF, derived from std::streambuf...">reflex::Input::dos_streambuf</a></code> to automatically convert <a class="el" href="index.html#regex-input-dosstreambuf">DOS CRLF newlines</a> by creating a <code>std::istream</code> for the specified <code><a class="el" href="classreflex_1_1_input_1_1dos__streambuf.html" title="Stream buffer for reflex::Input to read DOS files, replaces CRLF by LF, derived from std::streambuf...">reflex::Input::dos_streambuf</a></code>. Due to the extra layer introduced in the input processing stack, this option adds some overhead but requires no changes to the patterns and application code.</li>
<li>Rewrite the patterns to match both <code>\n</code> and <code>\r\n</code> to allow DOS CRLF input. This is option is fast to process input, but requires specialized patterns and the matched multi-line text will include <code>\r</code> (CR) characters that may need to be dealt with by the application code.</li>
</ol>
<p>To rewrite your patterns to support DOS CRLF matching:</p>
<ul>
<li>Replace <code>\n</code> in patterns by <code>\r?\n</code>.</li>
<li>Replace <code>.*</code> in patterns by <code>([^\n\r]|\r[^\n])*</code> to match any non-newline characters. Likewise replace <code>.+</code> by its longer version. Note that a single <code>.</code> can still be used in patterns but may match a <code>\r</code> just before a <code>\n</code> when a DOS CRLF is encountered.</li>
</ul>
<p>🔝 <a href="#">Back to contents</a></p>
<h2><a class="anchor" id="cr"></a>
Handling old Macintosh files containing CR newlines                       </h2>
<p>Old Macintosh OS file formats prior to Mac OS X use CR to end lines instead of LF. To automatically read and normalize files encoded in MacRoman containing CR as newlines, you can use the <code><a class="el" href="structreflex_1_1_input_1_1file__encoding.html#a5e9417330585b795cb0814de2f751f6b" title="Macintosh Roman with CR to LF translation. ">reflex::Input::file_encoding::macroman</a></code> file encoding format. This normalizes the input to UTF-8 and translates CR newlines to LF newlines. See <a class="el" href="index.html#regex-input-file">FILE encodings</a> for details.</p>
<p>Alternatively, you can define a custom code page to translate CR to LF without normalizing to UTF-8:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#define LF 10</span></div><div class="line"><span class="keyword">static</span> <span class="keyword">const</span> <span class="keywordtype">unsigned</span> <span class="keywordtype">short</span> CR2LF[256] = {</div><div class="line">    0,   1,   2,    3,    4,    5,    6,    7,    8,    9,   10,   11,   12,   LF,   14,   15,</div><div class="line">   16,  17,  18,   19,   20,   21,   22,   23,   24,   25,   26,   27,   28,   29,   30,   31,</div><div class="line">   32,  33,  34,   35,   36,   37,   38,   39,   40,   41,   42,   43,   44,   45,   46,   47,</div><div class="line">   48,  49,  50,   51,   52,   53,   54,   55,   56,   57,   58,   59,   60,   61,   62,   63,</div><div class="line">   64,  65,  66,   67,   68,   69,   70,   71,   72,   73,   74,   75,   76,   77,   78,   79,</div><div class="line">   80,  81,  82,   83,   84,   85,   86,   87,   88,   89,   90,   91,   92,   93,   94,   95,</div><div class="line">   96,  97,  98,   99,  100,  101,  102,  103,  104,  105,  106,  107,  108,  109,  110,  111,</div><div class="line">  112, 113, 114,  115,  116,  117,  118,  119,  120,  121,  122,  123,  124,  125,  126,  127,</div><div class="line">  128, 129, 130,  131,  132,  133,  134,  135,  136,  137,  138,  139,  140,  141,  142,  143,</div><div class="line">  144, 145, 146,  147,  148,  149,  150,  151,  152,  153,  154,  155,  156,  157,  158,  159,</div><div class="line">  160, 161, 162,  163,  164,  165,  166,  167,  168,  169,  170,  171,  172,  173,  174,  175,</div><div class="line">  176, 177, 178,  179,  180,  181,  182,  183,  184,  185,  186,  187,  188,  189,  190,  191,</div><div class="line">  192, 193, 194,  195,  196,  197,  198,  199,  200,  201,  202,  203,  204,  205,  206,  207,</div><div class="line">  208, 209, 210,  211,  212,  213,  214,  215,  216,  217,  218,  219,  220,  221,  222,  223,</div><div class="line">  224, 225, 226,  227,  228,  229,  230,  231,  232,  233,  234,  235,  236,  237,  238,  239,</div><div class="line">  240, 241, 242,  243,  244,  245,  246,  247,  248,  249,  250,  251,  252,  253,  254,  255</div><div class="line">};</div><div class="line"><a class="code" href="classreflex_1_1_input.html">reflex::Input</a> input(stdin, <a class="code" href="structreflex_1_1_input_1_1file__encoding.html#a76935cbd4a24d36af7e3ebdd5772aeb9">reflex::Input::file_encoding::custom</a>, CR2LF);</div></div><!-- fragment --><p>Then use the <code>input</code> object to read <code>stdin</code> or any other <code>FILE*</code>. See also <a class="el" href="index.html#regex-input-file">FILE encodings</a>.</p>
<p>🔝 <a href="#">Back to contents</a></p>
<h2><a class="anchor" id="lazy"></a>
Lazy repetitions                                                        </h2>
<p>Repetitions (<code>*</code>, <code>+</code>, and <code>{n,m}</code>) and the optional pattern (<code>?</code>) are greedy, unless marked with an extra <code>?</code> to make them lazy. Lazy repetitions are useless when the regex pattern after the lazy repetitions permits empty input. For example, <code>.*?a?</code> only matches one <code>a</code> or nothing at all, because <code>a?</code> permits an empty match.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="trigraphs"></a>
Lazy optional patterns and trigraphs                               </h2>
<p>This C/C++ trigraph problem work-around does not apply to lexer specifications that the <b><code>reflex</code></b> command converts while preventing trigraphs.</p>
<p>Trigraphs in C/C++ strings are special tripple-character sequences, beginning with two question marks and followed by a character that is translated. For example, <code>"x??(y|z)"</code> is translated to <code>"x[y|z)"</code>.</p>
<p>Fortunately, most C++ compilers ignore trigraphs unless in standard-conforming modes, such as <code>-ansi</code> and <code>-std=c++98</code>.</p>
<p>When using the lazy optional pattern <code>φ??</code> in a regex C/C++ string for pattern matching with one of the RE/flex matchers for example, use <code>φ?\?</code> instead, which the C/C++ compiler translates to <code>φ??</code>.</p>
<p>Otherwise, lazy optional pattern constructs will appear broken.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="switching"></a>
Repeately switching to the same input                              </h2>
<p>The state of the input object <code><a class="el" href="classreflex_1_1_input.html" title="Input character sequence class for unified access to sources of input text. ">reflex::Input</a></code> changes as the scanner's matcher consumes more input. If you switch to the same input again (e.g. with <code>in(i)</code> or <code>switch_stream(i)</code> for input source <code>i</code>), a portion of that input may end up being discarded as part of the matcher's internal buffer is flushed when input is assigned. Therefore, the following code will not work because stdin is flushed repeately:</p>
<div class="fragment"><div class="line">Lexer lexer(stdin);       <span class="comment">// a lexer that reads stdin</span></div><div class="line">lexer.in(stdin);          <span class="comment">// this is OK, nothing read yet</span></div><div class="line"><span class="keywordflow">while</span> (lexer.lex(stdin))  <span class="comment">// oops, assigning stdin again and again</span></div><div class="line">  std::cout &lt;&lt; <span class="stringliteral">&quot;we&#39;re not getting anywhere?&quot;</span> &lt;&lt; std::endl;</div></div><!-- fragment --><p>If you need to read a file or stream again, you must rewind it to the location in the file to start reading. Beware that <code>FILE*</code> input is checked against an UTF BOM at the start of a file, which means that you cannot reliably move to a location in the file to start reading when files are encoded in UTF-8 or UTF-16.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="flexlexer"></a>
Where is FlexLexer.h?                                              </h2>
<p>RE/flex uses its own header file <em><code><a class="el" href="flexlexer_8h.html" title="RE/flex Flex-compatible FlexLexer base class and Lex/Flex-compatible macros. ">reflex/flexlexer.h</a></code></em> for compatibility with Flex, instead of Flex file <code><a class="el" href="flexlexer_8h.html" title="RE/flex Flex-compatible FlexLexer base class and Lex/Flex-compatible macros. ">FlexLexer.h</a></code>. The latter is specific to Flex and cannot be used with RE/flex. You should not have to include <em><code><a class="el" href="flexlexer_8h.html" title="RE/flex Flex-compatible FlexLexer base class and Lex/Flex-compatible macros. ">FlexLexer.h</a></code></em> but if you do, use:</p>
<div class="fragment"><div class="line"><span class="preprocessor">#include &lt;<a class="code" href="flexlexer_8h.html">reflex/flexlexer.h</a>&gt;</span></div></div><!-- fragment --><p>The <code>FlexLexer</code> class defined in <em><code><a class="el" href="flexlexer_8h.html" title="RE/flex Flex-compatible FlexLexer base class and Lex/Flex-compatible macros. ">reflex/flexlexer.h</a></code></em> is the base class of the generated <code>yyFlexLexer</code> class. A name for the generated lexer class can be specified with option <code>−−lexer=NAME</code>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="readline"></a>
Interactive input with GNU readline                                 </h2>
<p>Option <code>-I</code> for interactive input generates a scanner that uses <code>fgetc()</code> to read input from a <code>FILE*</code> descriptor (stdin by default). Interactive input is made more user-friendly with the GNU readline library that provides basic line editing and a history mechanism.</p>
<p>To use <code>readline()</code> in your lexer, call <code>readline()</code> in your Lexer's constructor and in the <code>wrap()</code> method as follows:</p>
<div class="alt"> <div class="fragment"><div class="line">%top{</div><div class="line"><span class="preprocessor">  #include &lt;stdlib.h&gt;</span></div><div class="line"><span class="preprocessor">  #include &lt;stdio.h&gt;</span></div><div class="line"><span class="preprocessor">  #include &lt;readline/readline.h&gt;</span></div><div class="line"><span class="preprocessor">  #include &lt;readline/history.h&gt;</span></div><div class="line">}</div><div class="line"></div><div class="line">%<span class="keyword">class</span>{</div><div class="line">  <span class="keyword">const</span> <span class="keywordtype">char</span> *prompt;</div><div class="line">  <span class="comment">// we use wrap() to read the next line</span></div><div class="line">  <span class="keyword">virtual</span> <span class="keywordtype">bool</span> wrap() {</div><div class="line">    <span class="keywordflow">if</span> (line)</div><div class="line">    {</div><div class="line">      free((<span class="keywordtype">void</span>*)line);</div><div class="line">      line = readline(prompt);</div><div class="line">      <span class="keywordflow">if</span> (line != NULL)</div><div class="line">      {</div><div class="line">        <span class="keywordflow">if</span> (*line)</div><div class="line">          add_history(line);</div><div class="line">        linen.assign(line).push_back(<span class="charliteral">&#39;\n&#39;</span>);</div><div class="line">        in(linen);</div><div class="line">      }</div><div class="line">    }</div><div class="line">    <span class="comment">// wrap() == true means OK: wrapped after EOF</span></div><div class="line">    <span class="keywordflow">return</span> line != NULL;</div><div class="line">  }</div><div class="line">  <span class="comment">// the line returned by readline() without \n</span></div><div class="line">  <span class="keywordtype">char</span> *line;</div><div class="line">  <span class="comment">// the line with \n appended</span></div><div class="line">  std::string linen;</div><div class="line">}</div><div class="line"></div><div class="line">%init{</div><div class="line">  prompt = NULL;</div><div class="line">  line = readline(prompt);</div><div class="line">  <span class="keywordflow">if</span> (line != NULL)</div><div class="line">  {</div><div class="line">    <span class="keywordflow">if</span> (*line)</div><div class="line">      add_history(line);</div><div class="line">    linen.assign(line).push_back(<span class="charliteral">&#39;\n&#39;</span>);</div><div class="line">  }</div><div class="line">  in(linen);</div><div class="line">}</div></div><!-- fragment --> </div><p>With option <code>−−flex</code> you will need to replace <code>wrap()</code> by a Flex-like <code>yywrap()</code> and change it to return 0 on success:</p>
<div class="alt"> <div class="fragment"><div class="line"><span class="comment">// we use yywrap() in Flex mode to read the next line</span></div><div class="line"><span class="keyword">virtual</span> <span class="keywordtype">int</span> yywrap() {</div><div class="line">  <span class="keywordflow">if</span> (line)</div><div class="line">  {</div><div class="line">    free((<span class="keywordtype">void</span>*)line);</div><div class="line">    line = readline(prompt);</div><div class="line">    <span class="keywordflow">if</span> (line != NULL)</div><div class="line">    {</div><div class="line">      <span class="keywordflow">if</span> (*line)</div><div class="line">        add_history(line);</div><div class="line">      linen.assign(line).push_back(<span class="charliteral">&#39;\n&#39;</span>);</div><div class="line">      in(linen);</div><div class="line">    }</div><div class="line">  }</div><div class="line">  <span class="comment">// yywrap() == 0 means OK: wrapped after EOF</span></div><div class="line">  <span class="keywordflow">return</span> line != NULL ? 0 : 1;</div><div class="line">}</div></div><!-- fragment --> </div><p>The rules can be matched as usual, where <code>\n</code> matches the end of a line, for example:</p>
<div class="alt"> <div class="fragment"><div class="line">%%</div><div class="line"></div><div class="line">.+  echo(); <span class="comment">// ECHO the entire line</span></div><div class="line">\n  echo(); <span class="comment">// ECHO end of the line</span></div><div class="line"></div><div class="line">%%</div></div><!-- fragment --> </div><p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="link-errors"></a>
Undefined symbols and link errors                                </h2>
<p>Some hints when dealing with undefined symbols and link errors when building RE/flex applications:</p>
<ul>
<li><p class="startli">Compilation requires <code>libreflex</code> which is linked using compiler option <code>-lreflex</code>: </p><pre class="fragment">c++ ... -lreflex
</pre><p class="startli">If <code>libreflex</code> was not installed on your system then header files cannot be found and linking with <code>-lreflex</code> fails. Instead, you can specify the include path and link the library with: </p><pre class="fragment">c++ -I&lt;path&gt;/reflex/include ... -L&lt;path&gt;/reflex/lib -lreflex
</pre><p class="startli">where <code>&lt;path&gt;</code> is the directory path to the top directory of the downloaded RE/flex package.</p>
</li>
<li>When using an IDE such as Visual Studio C++, compile all of the <em>.cpp</em> source code files located in the <em><code>reflex/lib</code></em> and the <em><code>reflex/unicode</code></em> directories of the RE/flex download package. The header files are located in the <em><code>reflex/include/reflex</code></em> directory.</li>
<li>When Boost.Regex is used as a matcher engine, also link <code>libboost_regex</code> (or a variant, e.g. <code>libboost_regex-mt</code>, depending on your Boost installation configuration): <pre class="fragment">c++ ... -lreflex -lboost_regex
</pre></li>
<li>When PCRE2 is used as a matcher engine, also link <code>libpcre2-8</code>: <pre class="fragment">c++ ... -lreflex -lpcre2-8
</pre></li>
<li>If you get compilation errors with the <code>std::regex</code> matching engine, you should compile the source code as C++11: <pre class="fragment">c++ -std=c++11 ... -lreflex
</pre></li>
<li>When generating scanners with the <b><code>reflex</code></b> tool, the generated <em><code>lex.yy.cpp</code></em> lexer logic should be compiled and linked with your application. We use <b><code>reflex</code></b> option <code>−−header-file</code> to generate <em><code>lex.yy.h</code></em> with the lexer class to include in the source code of your lexer application.</li>
</ul>
<p>🔝 <a href="#">Back to contents</a></p>
<h2><a class="anchor" id="linking"></a>
Minimized library                                                    </h2>
<p>RE/flex scanners generated by <b><code>reflex</code></b> can be linked against a minimized version of the RE/flex library <code>libreflexmin</code>: </p><pre class="fragment">  c++ ... -lreflexmin
</pre><p>The regex converters and the Unicode tables that take up space are excluded from this minimized library.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h2><a class="anchor" id="msvc"></a>
MSVC++ compiler bug                                                     </h2>
<p>Some MSVC++ compilers may cause problems with C++11 range-based loops. When a matcher object is constructed in a range-based loop it is destroyed before the first loop iteration. This means that the following example crashes:</p>
<div class="fragment"><div class="line"><span class="keywordflow">for</span> (<span class="keyword">auto</span>&amp; match : <a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a>(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>).find)</div><div class="line">  std::cout &lt;&lt; match.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>Instead, we should write the following:</p>
<div class="fragment"><div class="line"><a class="code" href="classreflex_1_1_boost_matcher.html">reflex::BoostMatcher</a> matcher(<span class="stringliteral">&quot;\\w+&quot;</span>, <span class="stringliteral">&quot;How now brown cow.&quot;</span>);</div><div class="line"><span class="keywordflow">for</span> (<span class="keyword">auto</span>&amp; match : matcher.find)</div><div class="line">  std::cout &lt;&lt; match.text() &lt;&lt; std::endl;</div></div><!-- fragment --><p>🔝 <a href="#">Back to contents</a></p>
<h1><a class="anchor" id="bugs"></a>
Bugs                                                                    </h1>
<p>Please report bugs as RE/flex GitHub <a href="https://github.com/Genivia/RE-flex/issues">issues</a>.</p>
<p>Please make sure to install the RE/flex library you download and remove old versions of RE/flex or otherwise prevent mixing old with new versions. Mixing old with new versions may cause problems. For example, when new versions of RE/flex header files are imported into your project but an old RE/flex library version is still linked with your code, the library may likely misbehave.</p>
<p>🔝 <a href="#">Back to contents</a></p>
<h1><a class="anchor" id="download"></a>
Installing RE/flex                                                  </h1>
<p>Download RE/flex from <a href="https://sourceforge.net/projects/re-flex">SourceForge</a> or visit the RE/flex GitHub <a href="https://github.com/Genivia/RE-flex">repository</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<h1><a class="anchor" id="license"></a>
License and copyright                                                </h1>
<p>RE/flex software is released under the BSD-3 license. All parts of the software have reasonable copyright terms permitting free redistribution. This includes the ability to reuse all or parts of the RE/flex source tree.</p>
<blockquote class="doxtable">
<p>Copyright (c) 2016, Robert van Engelen, Genivia Inc. All rights reserved.</p>
<p>Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:</p>
<p>(1) Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.</p>
<p>(2) Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.</p>
<p>(3) The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission.</p>
<p>THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. </p>
</blockquote>
<p>The Free Software Foundation maintains a <a href="http://directory.fsf.org/wiki/License:BSD_3Clause">BSD-3 License Wiki</a>.</p>
<p>🔝 <a href="#">Back to table of contents</a></p>
<p>Copyright (c) 2016-2020, Robert van Engelen, Genivia Inc. All rights reserved. </p>
</div></div><!-- contents -->
<hr class="footer">
<address class="footer"><small>
Converted on Thu May 14 2020 13:20:19 by <a target="_blank" href="http://www.doxygen.org/index.html">Doxygen</a> 1.8.11</small></address>
<br>
<div style="height: 246px; background: #DBDBDB;">
</body>
</html>
